Product Code Database
Example Keywords: underclothes -super $73-132
barcode-scavenger
   » » Wiki: Newline
Tag Wiki 'Newline'.
Tag
Newline (frequently called line ending, end of line ( EOL), line feed, or line break) is a control character in a character encoding specification, like e.g. . It is used to signify the end of a line of text and the start of a new one. Text editors set this special character when pressing the .

When displaying (or printing) a , this control character causes the text editor to show the following characters in a new line.


Wall of text
The concepts of line feed ( LF) and ( CR) are closely associated, and can be considered either separately or together. In the physical media of and printers, two axes of motion, "down" and "across", are needed to create a new line on the page. Although the design of a machine (typewriter or printer) must consider them separately, the abstract logic of software can combine them together as one event. This is why a newline in character encoding can be defined as LF and CR combined into one (commonly called CR+LF or CRLF).

Some character sets provide a separate newline character code. , for example, provides an NL character code in addition to the CR and LF codes. , in addition to providing the CR and LF control codes, also provides a "next line" (NEL) control code, as well as control codes for "line separator" and "paragraph separator" markers.

Two ways to view newlines, both of which are , are that newlines either separate lines or that they terminate lines. If a newline is considered a separator, there will be no newline after the last line of a file. Some programs have problems processing the last line of a file if it is not terminated by a newline. On the other hand, programs that expect newline to be used as a separator will interpret a final newline as starting a new (empty) line. Conversely, if a newline is considered a terminator, all text lines including the last are expected to be terminated by a newline. If the final character sequence in a text file is not a newline, the final line of the file may be considered to be an improper or incomplete text line, or the file may be considered to be improperly truncated.

In text intended primarily to be read by humans using software which implements the feature, a newline character typically only needs to be stored if a line break is required independent of whether the next word would fit on the same line, such as between and in vertical lists. Therefore, in the logic of and most , newline is used as a paragraph break and is known as a "hard return", in contrast to "soft returns" which are dynamically created to implement word wrapping and are changeable with each display instance. In many applications a separate control character called "manual line break" exists for forcing line breaks inside a single paragraph. The for the control character for a hard return is usually a (¶), and for the manual line break is usually a carriage return arrow (↵).


Representations in different character encoding specifications
Software applications and usually represent a newline with one or two control characters:

, and systems (, , , AIX, , etc.), , , , and othersASCII
, Microsoft Windows, (, , etc.), DEC TOPS-10, RT-11, CP/M, MP/M, OS/2, , , , and most other early non-Unix and non-IBM operating systems
Commodore 8-bit machines (C64), , , TRS-80, Apple II family, Oberon, the classic Mac OS, MIT and OS-9
pre-POSIX implementation (version < 4)
and spooled text output.+
Atari 8-bit machines
(155 in decimal)
IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400), OSEBCDIC
ZX80 and ZX81 (Home computers from Sinclair Research Ltd)used a specific non-ASCII character setNEWLINE

  • systems—mainly IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400)—use (New Line, 0x15)IBM System/360 Reference Data Card, Publication GX20-1703, IBM Data Processing Division, White Plains, NY as the character combining the functions of line-feed and carriage-return. The equivalent UNICODE character is called (Next Line). EBCDIC also has control characters called and , but the numerical value of (0x25) differs from the one used by ASCII (0x0A). Additionally, some EBCDIC variants also use but assign a different numeric code to the character.
  • Operating systems for the CDC 6000 series defined a newline as two or more zero-valued six-bit characters at the end of a 60-bit word. Some configurations also defined a zero-valued character as a colon character, with the result that multiple colons could be interpreted as a newline depending on position.
  • RSX-11 and use a record-based file system, which stores text files as one record per line. In most file formats, no line terminators are actually stored, but the Record Management Services facility can transparently add a terminator to each line when it is retrieved by an application. The records themselves could contain the same line terminator characters, which could either be considered a feature or a nuisance depending on the application. RMS not only stored records, but also stored metadata about the record separators in different bits for the file to complicate matters even more (since files could have fixed length records, records that were prefixed by a count or records that were terminated by a specific character). The bits weren't generic, so while they could specify that or or even was the line terminator, it couldn't substitute some other code.
  • Fixed line length was used by some early mainframe operating systems. In such a system, an implicit end-of-line was assumed every 72 or 80 characters, for example. No newline character was stored. If a file was imported from the outside world, lines shorter than the line length had to be padded with spaces, while lines longer than the line length had to be truncated. This mimicked the use of , on which each line was stored on a separate card, usually with 80 columns on each card, often with sequence numbers in columns 73–80. Many of these systems added a carriage control character to the start of the next record; this could indicate whether the next record was a continuation of the line started by the previous record, or a new line, or should overprint the previous line (similar to a CR). Often this was a normal printing character such as "#" that thus could not be used as the first character in a line. Some early line printers interpreted these characters directly in the records sent to them.

Most textual protocols (including , SMTP, FTP, IRC, and many others) mandate the use of ASCII + (, ) on the protocol level, but recommend that tolerant applications recognize lone (, ) as well. Despite the dictated standard, many applications erroneously use the C newline escape sequence () instead of the correct combination of carriage return escape and newline escape sequences (+) (see section Newline in programming languages below). This accidental use of the wrong escape sequences leads to problems when trying to communicate with systems adhering to the stricter interpretation of the standards instead of the suggested tolerant interpretation. One such intolerant system is the mail transfer agent that actively refuses to accept messages from systems that send bare instead of the required +.

The standard Internet Message Format for eMail states: CR and LF MUST only occur together as CRLF; they MUST NOT appear independently in the body.

has a feature to transform newlines between CR+LF and the native encoding of the system (e.g., LF only) when transferring text files. This feature must not be used on binary files. Often binary files and text files are recognised by checking their filename extension; most command-line FTP clients have an explicit command to switch between binary and text mode transfers.


Unicode
The standard defines a number of characters that conforming applications should recognize as line terminators:

: Line Feed,
: ,
: ,
: ,
+: () followed by ()
: Next Line,
: Line Separator,
: Paragraph Separator,

This may seem overly complicated compared to an approach such as converting all line terminators to a single character, for example . However, Unicode was designed to preserve all information when converting a text file from any existing encoding to Unicode and back. Therefore, Unicode should contain characters included in existing encodings. is included in with code (0x15). is also a control character in the C1 control set. As such, it is defined by ECMA 48, and recognized by encodings compliant with ISO/IEC 2022 (which is equivalent to ECMA 35). C1 control set is also compatible with ISO-8859-1. The approach taken in the Unicode standard allows round-trip transformation to be information-preserving while still enabling applications to recognize all possible types of line terminators.

Recognizing and using the newline codes greater than 0x7F is not often done. They are multiple bytes in UTF-8, and the code for NEL has been used as the ('…') character in Windows-1252. For instance:

  • accepts LS and PS as line breaks, but considers U+0085 (NEL) white space, not a line break.
  • Windows 10 does not treat any of NEL, LS, or PS as line-break in the default text editor Notepad
  • On Linux, a popular editor, , treats LS and PS as newlines but does not for NEL.
  • no longer recognizes them as special, in order to be compatible with .
  • JSON treats LS and PS inside a String as letter while ECMAScript/Javascript treats them as new line and shows an error

The Unicode characters U+2424 (SYMBOL FOR NEWLINE, ␤), U+23CE (RETURN SYMBOL, ⏎), U+240D (SYMBOL FOR CARRIAGE RETURN, ␍) and U+240A (SYMBOL FOR LINE FEED, ␊) are intended for presenting a user-visible character to the reader of the document, and are thus not recognized themselves as a newline.


Escape sequences
An is a combination of characters which represents no text; instead of being displayed (as text) it is supposed to be intercepted by the program and a special function is supposed to be performed. Escape sequences are also used to handle (set, search, replace, etc.) special characters.


In programming languages
To facilitate the creation of programs, programming languages provide some abstractions to deal with the different types of newline sequences used in different environments.

The C programming language provides the (newline) and (carriage return). However, these are not required to be equivalent to the ASCII and control characters. The C standard only guarantees two things:

  1. Each of these escape sequences maps to a unique implementation-defined number that can be stored in a single value.
  2. When writing to a file, device node, or socket/fifo in text mode, is transparently translated to the native newline sequence used by the system, which may be longer than one character. When reading in text mode, the native newline sequence is translated back to . In binary mode, no translation is performed, and the internal representation produced by is output directly.

On Unix platforms, where C originated, the native newline sequence is ASCII (), so was simply defined to be that value. With the internal and external representation being identical, the translation performed in text mode is a , and Unix has no notion of text mode or binary mode. This has caused many programmers who developed their software on Unix systems simply to ignore the distinction completely, resulting in code that is not portable to different platforms.

The C library function fgets() is best avoided in binary mode because any file not written with the Unix newline convention will be misread. Also, in text mode, any file not written with the system's native newline sequence (such as a file created on a Unix system, then copied to a Windows system) will be misread as well.

Another common problem is the use of when communicating using an Internet protocol that mandates the use of ASCII + for ending lines. Writing to a text mode stream works correctly on Windows systems, but produces only on Unix, and something completely different on more exotic systems. Using in binary mode is slightly better.

Many languages, such as C++, , and Haskell provide the same interpretation of as C.

Java, , and Python provide the sequence (for ASCII +). In contrast to C, these are guaranteed to represent the values and , respectively.

The Java I/O libraries do not transparently translate these into platform-dependent newline sequences on input or output. Instead, they provide functions for writing a full line that automatically add the native newline sequence, and functions for reading lines that accept any of , , or + as a line terminator (see ). The method can be used to retrieve the underlying line separator.

Example:

  String eol = System.lineSeparator();
  String lineColor = "Color: Red" + eol;
     

Python permits "Universal Newline Support" when opening a file for reading, when importing modules, and when executing a file.

Some languages have created special variables, constants, and to facilitate newlines during program execution. In some languages such as and , are required to perform escape substitution for all escape sequences, including and . In PHP, to avoid portability problems, newline sequences should be issued using the PHP_EOL constant.

Example in C#:

  string eol = Environment.NewLine;
  string lineColor = "Color: Red" + eol;
     

  string eol2 = "\n";
  string lineColor2 = "Color: Blue" + eol2;
     


Frequent issues
Even though the control characters are unambiguously defined in the corresponding character encoding table used by a text file, there still is an issue: there are different conventions to set and display a line break.

To denote a single line break, programs use line feed, whose hexadecimal value in ASCII is :%s/}/}\r\t/g, while most programs common to and Microsoft Windows use carriage return+tabulator, whose hexadecimal value in ASCII is line feed. In ASCII, is a distinct control character.

The different newline conventions cause text files that have been transferred between systems of different types to be displayed incorrectly.

Text in files created with programs which are common on or classic Mac OS, appear as a single long line on most programs common to and Microsoft Windows because these do not display a single 0a as a line break.

Conversely, when viewing a file originating from a Windows computer on a Unix-like system, the extra may be displayed as a second line break, as or at the end of each line.

Furthermore, programs other than text editors may not accept a file, e.g. some configuration file, encoded using the foreign newline convention, as a valid file.

The problem can be hard to spot because some programs handle the foreign newlines properly while others do not. For example, a may fail with obscure syntax errors even though the source file looks correct when displayed on the console or in an . On a Unix-like system, the command will send the file to stdout (normally the terminal) and make the visible, which can be useful for debugging. Modern text editors generally recognize all flavours of + newlines and allow users to convert between the different standards. are usually also capable of displaying text files and websites which use different types of newlines.

Even if a program supports different newline conventions, these features are often not sufficiently labeled, described, or documented. Typically a menu or combo-box enumerating different newline conventions will be displayed to users without an indication if the selection will re-interpret, temporarily convert, or permanently convert the newlines. Some programs will implicitly convert on open, copy, paste, or save—often inconsistently.

The File Transfer Protocol can automatically convert newlines in files being transferred between with different newline representations when the transfer is done in "ASCII mode". However, transferring binary files in this mode usually has disastrous results: any occurrence of the newline byte sequence—which does not have line terminator semantics in this context, but is just part of a normal sequence of bytes—will be translated to whatever newline representation the other system uses, effectively the file. FTP clients often employ some heuristics (for example, inspection of filename extensions) to automatically select either binary or ASCII mode, but in the end it is up to users to make sure their files are transferred in the correct mode. If there is any doubt as to the correct mode, binary mode should be used, as then no files will be altered by FTP, though they may display incorrectly.


Conversion utilities
are often used for converting a text file between different newline formats; most modern editors can read and write files using at least the different ASCII / conventions. The standard Windows editor Notepad is not one of them (although and the are).

Editors are often unsuitable for converting larger files. For larger files (on Windows NT/2000/XP) the following command is often used: D:\>TYPE unix_file | FIND /V "" > dos_file On many systems, the (sometimes named or ) and (sometimes named or ) utilities are used to translate between ASCII + (DOS/Windows) and (Unix) newlines. Different versions of these commands vary slightly in their syntax. However, the command is available on virtually every system and can be used to perform arbitrary replacement operations on single characters. A DOS/Windows text file can be converted to Unix format by simply removing all ASCII characters with

$ tr -d '\r' < ''inputfile'' > ''outputfile''
     
or, if the text has only newlines, by converting all newlines to with
$ tr '\r' '\n' < ''inputfile'' > ''outputfile''
     

The same tasks are sometimes performed with , , or in if the platform has a Perl interpreter: $ awk '{sub("$","\r\n"); printf("%s",$0);}' inputfile > outputfile # UNIX to DOS (adding CRs on Linux and BSD based OS that haven't GNU extensions) $ awk '{gsub("\r",""); print;}' inputfile > outputfile # DOS to UNIX (removing CRs on Linux and BSD based OS that haven't GNU extensions) $ sed -e 's/$/\r/' inputfile > outputfile # UNIX to DOS (adding CRs on Linux based OS that use GNU extensions) $ sed -e 's/\r$//' inputfile > outputfile # DOS to UNIX (removing CRs on Linux based OS that use GNU extensions) $ cat inputfile | tr -d "\r" > outputfile # DOS to UNIX (removing CRs using tr(1). Not Unicode compliant.) $ perl -pe 's/\r?\n|\r/\r\n/g' inputfile > outputfile # Convert to DOS $ perl -pe 's/\r?\n|\r/\n/g' inputfile > outputfile # Convert to UNIX $ perl -pe 's/\r?\n|\r/\r/g' inputfile > outputfile # Convert to old Mac To identify what type of line breaks a text file contains, the file command can be used. Moreover, the editor Vim can be convenient to make a file compatible with the Windows notepad text editor. For example:

$ file myfile.txt
myfile.txt: ASCII English text
$ vim myfile.txt
     
within vim
:set fileformat=dos
:wq
     
$ file myfile.txt
myfile.txt: ASCII English text, with CRLF line terminators
     
The following commands echo the filename (in this case myfile.txt) to the command line if the file is of the specified style: $ grep -PL $'\r\n' myfile.txt # show UNIX style file (LF terminated) $ grep -Pl $'\r\n' myfile.txt # show DOS style file (CRLF terminated) For systems with egrep (extended grep) such as () based systems and many other systems, these commands can be used: $ egrep -L $'\r\n' myfile.txt # show UNIX style file (LF terminated) $ egrep -l $'\r\n' myfile.txt # show DOS style file (CRLF terminated) The above grep commands work under systems or in under Windows. Note that these commands make some assumptions about the kinds of files that exist on the system (specifically it's assuming only Unix and DOS-style files—no Mac OS 9-style files).

This technique is often combined with find to list files recursively. For instance, the following command checks all "regular files" (e.g. it will exclude directories, symbolic links, etc.) to find all Unix-style files in a directory tree, starting from the current directory (.), and saves the results in file unix_files.txt, overwriting it if the file already exists: $ find . -type f -exec grep -PL '\r\n' {} \; > unix_files.txt This example will find C files and convert them to LF style line endings: $ find -name '*.ch' -exec fromdos {} \; The file command also detects the type of EOL used: $ file myfile.txt myfile.txt: ASCII text, with CRLF line terminators Other tools permit the user to visualise the EOL characters: $ od -a myfile.txt $ cat -e myfile.txt $ hexdump -c myfile.txt dos2unix, unix2dos, mac2unix, unix2mac, mac2dos, dos2mac can perform conversions. The flip command is often used.


History
In the mid-1800s, long before the advent of and teletype machines, operators or invented and used Morse code prosigns to encode white space text formatting in formal written text messages. In particular the Morse prosign represented by the concatenation of two literal textual Morse code "A" characters sent without the normal inter-character spacing is used in Morse code to encode and indicate a new line in a formal text message.

Later in the age of modern standardized character set control codes were developed to aid in white space text formatting. ASCII was developed simultaneously by the International Organization for Standardization (ISO) and the American Standards Association (ASA), the latter being the predecessor organization to American National Standards Institute (ANSI). During the period of 1963 to 1968, the ISO draft standards supported the use of either + or alone as a newline, while the ASA drafts supported only +.

The sequence + was in common use on many early computer systems that had adopted Teletype machines, typically a Teletype Model 33 ASR, as a console device, because this sequence was required to position those printers at the start of a new line. The separation of newline into two functions concealed the fact that the print head could not return from the far right to the beginning of the next line in one-character time. That is why the sequence was always sent with the first. A character printed after a CR would often print as a smudge, on-the-fly in the middle of the page, while it was still moving the carriage back to the first position. "The solution was to make the newline two characters: CR to move the carriage to column one, and LF to move the paper up."

(2018). 9780735710016, Sams. .
In fact, it was often necessary to send extra characters (extraneous CRs or NULs, which are ignored) to give the print head time to move to the left margin. Even many early video displays required multiple character times to the display.

On these systems, text was often routinely composed to be compatible with these printers, since the concept of hiding such hardware details from the application was not yet well developed; applications had to talk directly to the Teletype machine and follow its conventions. Most minicomputer systems from DEC used this convention. CP/M used it as well, to print on the same terminals that minicomputers used. From there (1981) adopted CP/M's + in order to be compatible, and this convention was inherited by Microsoft's later Windows operating system.

The operating system began development in 1964 and used alone as its newline. Multics used a device driver to translate this character to whatever sequence a printer needed (including extra padding characters), and the single byte was much more convenient for programming. What now seems a more obvious choice of was not used, as a plain provided the useful function of overprinting one line with another to create boldface and effects, and thus it was useful to not translate it. Perhaps more importantly, the use of alone as a line terminator had already been incorporated into drafts of the eventual ISO/IEC 646 standard. followed the Multics practice, and later systems followed Unix.


Reverse and partial line feeds
, (+008D REVERSE LINE FEED, ISO/IEC 6429 8D, decimal 141) is used to move the printing position back one line (by reverse feeding the paper, or by moving a display cursor up one line) so that other characters may be printed over existing text. This may be done to make them bolder, or to add underlines, strike-throughs or other characters such as .

Similarly, (+008B PARTIAL LINE FORWARD, decimal 139) and (+008C PARTIAL LINE BACKWARD, decimal 140) can be used to advance or reverse the text printing position by some fraction of the vertical line spacing (typically, half). These can be used in combination for subscripts (by advancing and then reversing) and superscripts (by reversing and then advancing), and may also be useful for printing diacritics.


See also


External links

Page 1 of 1
1

Account

Social:
Pages:  ..   .. 
Items:  .. 

Navigation

General: Atom Feed Atom Feed  .. 
Help:  ..   .. 
Category:  ..   .. 
Media:  ..   .. 
Posts:  ..   ..   .. 

Statistics

Page:  .. 
Summary:  .. 
1 Tags
10/10 Page Rank
5 Page Refs
1s Time