Introduction
Limitations
Feature set
Typedefs
Comment on naming
Class template endian
Synopsis
Members
FAQ
Example
Design
Experience
Acknowledgements
The boost/integer/endian.hpp header provides integer-like byte-holder binary types with explicit control over byte order, value type, size, and alignment. Typedefs provide easy-to-use names for common configurations.
These types provide portable byte-holders for integer data, independent of particular computer architectures. Use cases almost always involve I/O, either via files or network connections. Although portability is the primary motivation, these integer byte-holders may also be used to reduce memory use, file size, or network activity since they provide binary integer sizes not otherwise available.
Such integer byte-holder types are traditionally called endian types. See the Wikipedia for a full exploration of endianness, including definitions of big endian and little endian.
Boost endian integers provide the same full set of C++ assignment,
arithmetic, and relational operators as C++ standard integral types, with
the standard semantics, plus operators <<
and >>
for
stream insertion and extraction.
Unary arithmetic operators are +
, -
, ~
,
!
, prefix and postfix --
and ++
. Binary
arithmetic operators are +
, +=
, -
,
-=
, *
, *=
, /
, /=
,
%/ %=
, &
, &=
, |
, |=
,
^
, ^=
, <<
, <<=
, >>
,
>>=
. Binary relational operators are ==
, !=
,
<
, <=
, >
, >=
.
Automatic conversion are provided to and from the underlying integer value type.
Requires <climits>
CHAR_BIT == 8
. If CHAR_BIT
is some other value, compilation will result in an #error
. This
restriction is in place because the design, implementation, testing, and
documentation has only considered issues related to 8-bit bytes, and there have
been no real-world use cases presented for other sizes.
One class template is provided:
template <endianness E, typename T, std::size_t n_bytes, alignment A> class endian;
Sixty typedefs, such as big32_t
, provide convenient naming
conventions for common use cases:
Name Endianness Sign Sizes in bits (n) Alignment big
n_t
big
signed 8,16,24,32,40,48,56,64 unaligned
ubig
n_t
big
unsigned 8,16,24,32,40,48,56,64 unaligned
little
n_t
little
signed 8,16,24,32,40,48,56,64 unaligned
ulittle
n_t
little
unsigned 8,16,24,32,40,48,56,64 unaligned
native
n_t
native
signed 8,16,24,32,40,48,56,64 unaligned
unative
n_t
native
unsigned 8,16,24,32,40,48,56,64 unaligned
aligned_big
n_t
big
signed 16,32,64 aligned
aligned_ubig
n_t
big
unsigned 16,32,64 aligned
aligned_little
n_t
little
signed 16,32,64 aligned
aligned_ulittle
n_t
little
unsigned 16,32,64 aligned
The unaligned types do not cause compilers to insert padding bytes in classes and structs. This is an important characteristic that can be exploited to minimize wasted space in files and network transmissions.
Warning: Code that uses aligned types is inherently non-portable because alignment requirements vary between hardware architectures and because alignment may be affected by compiler switches or pragmas. Furthermore, aligned types are only available on architectures with 16, 32, and 64-bit integer types.
Note: One-byte big-endian, little-endian, and native-endian types provide identical functionality. All three names are provided to improve code readability and searchability.
When first exposed to endian types, programmers often fit them into a mental model
based on the <cstdint>
types. Using that model, it is natural to
expect a 56-bit big-endian signed integer to be named int_big56_t
rather than big56_t
.
As experience using these type grows, the realization creeps in that they are lousy arithmetic integers - they are really byte holders that for convenience support arithmetic operations - and that for use in internal interfaces or anything more than trivial arithmetic computations it is far better to convert values of these endian types to traditional integer types.
That seems to lead to formation of a new mental model specific to endian byte-holder types. In that model, the endianness
is the key feature, and the integer aspect is downplayed.
Once that mental transition is made, a name like big56_t
is a good
reflection of the mental model
endian
An endian is an integer byte-holder with user-specified endianness, value type, size, and alignment. The usual operations on integers are supplied.
namespace boost { namespace integer { enum endianness { big, little, native }; enum alignment { unaligned, aligned }; template <endianness E, typename T, std::size_t n_bits, alignment A> class endian : integer_cover_operators< endian<E, T, n_bits, A>, T > { public: typedef T value_type; endian(); endian(T v); operator T() const; }; // unaligned big endian signed integer types typedef endian< big, int_least8_t, 8 > big8_t; typedef endian< big, int_least16_t, 16 > big16_t; typedef endian< big, int_least32_t, 24 > big24_t; typedef endian< big, int_least32_t, 32 > big32_t; typedef endian< big, int_least64_t, 40 > big40_t; typedef endian< big, int_least64_t, 48 > big48_t; typedef endian< big, int_least64_t, 56 > big56_t; typedef endian< big, int_least64_t, 64 > big64_t; // unaligned big endian unsigned integer types typedef endian< big, uint_least8_t, 8 > ubig8_t; typedef endian< big, uint_least16_t, 16 > ubig16_t; typedef endian< big, uint_least32_t, 24 > ubig24_t; typedef endian< big, uint_least32_t, 32 > ubig32_t; typedef endian< big, uint_least64_t, 40 > ubig40_t; typedef endian< big, uint_least64_t, 48 > ubig48_t; typedef endian< big, uint_least64_t, 56 > ubig56_t; typedef endian< big, uint_least64_t, 64 > ubig64_t; // unaligned little endian signed integer types typedef endian< little, int_least8_t, 8 > little8_t; typedef endian< little, int_least16_t, 16 > little16_t; typedef endian< little, int_least32_t, 24 > little24_t; typedef endian< little, int_least32_t, 32 > little32_t; typedef endian< little, int_least64_t, 40 > little40_t; typedef endian< little, int_least64_t, 48 > little48_t; typedef endian< little, int_least64_t, 56 > little56_t; typedef endian< little, int_least64_t, 64 > little64_t; // unaligned little endian unsigned integer types typedef endian< little, uint_least8_t, 8 > ulittle8_t; typedef endian< little, uint_least16_t, 16 > ulittle16_t; typedef endian< little, uint_least32_t, 24 > ulittle24_t; typedef endian< little, uint_least32_t, 32 > ulittle32_t; typedef endian< little, uint_least64_t, 40 > ulittle40_t; typedef endian< little, uint_least64_t, 48 > ulittle48_t; typedef endian< little, uint_least64_t, 56 > ulittle56_t; typedef endian< little, uint_least64_t, 64 > ulittle64_t; // unaligned native endian signed integer types typedef endian< native, int_least8_t, 8 > native8_t; typedef endian< native, int_least16_t, 16 > native16_t; typedef endian< native, int_least32_t, 24 > native24_t; typedef endian< native, int_least32_t, 32 > native32_t; typedef endian< native, int_least64_t, 40 > native40_t; typedef endian< native, int_least64_t, 48 > native48_t; typedef endian< native, int_least64_t, 56 > native56_t; typedef endian< native, int_least64_t, 64 > native64_t; // unaligned native endian unsigned integer types typedef endian< native, uint_least8_t, 8 > unative8_t; typedef endian< native, uint_least16_t, 16 > unative16_t; typedef endian< native, uint_least32_t, 24 > unative24_t; typedef endian< native, uint_least32_t, 32 > unative32_t; typedef endian< native, uint_least64_t, 40 > unative40_t; typedef endian< native, uint_least64_t, 48 > unative48_t; typedef endian< native, uint_least64_t, 56 > unative56_t; typedef endian< native, uint_least64_t, 64 > unative64_t; // These types only present if platform has exact size integers: // aligned big endian signed integer types typedef endian< big, int16_t, 16, aligned > aligned_big16_t; typedef endian< big, int32_t, 32, aligned > aligned_big32_t; typedef endian< big, int64_t, 64, aligned > aligned_big64_t; // aligned big endian unsigned integer types typedef endian< big, uint16_t, 16, aligned > aligned_ubig16_t; typedef endian< big, uint32_t, 32, aligned > aligned_ubig32_t; typedef endian< big, uint64_t, 64, aligned > aligned_ubig64_t; // aligned little endian signed integer types typedef endian< little, int16_t, 16, aligned > aligned_little2_t; typedef endian< little, int32_t, 32, aligned > aligned_little4_t; typedef endian< little, int64_t, 64, aligned > aligned_little8_t; // aligned little endian unsigned integer types typedef endian< little, uint16_t, 16, aligned > aligned_ulittle2_t; typedef endian< little, uint32_t, 32, aligned > aligned_ulittle4_t; typedef endian< little, uint64_t, 64, aligned > aligned_ulittle8_t; // aligned native endian typedefs are not provided because // <cstdint> types are superior for this use case } // namespace integer } // namespace boost
endian();
Effects: Constructs an object of type
endian<E, T, n_bits, A>
.
endian(T v);
Effects: Constructs an object of type
endian<E, T, n_bits, A>
.Postcondition:
x == v,
wherex
is the constructed object.
operator T() const;
Returns: The current value stored in
*this
, converted tovalue_type
.
All other operators on endian objects are forwarded to the equivalent
operator on value_type
.
Why bother with binary I/O? Why not just use C++ Standard Library stream inserters and extractors? Using binary rather than character representations can be more space efficient, with a side benefit of faster I/O. CPU time is minimized because conversions to and from string are eliminated. Furthermore, binary integers are fixed size, and so fixed-size disk records are possible, easing sorting and allowing direct access. Disadvantages, such as the inability to use text utilities on the resulting files, limit usefulness to applications where the binary I/O advantages are paramount.
Why bother with endian types? Primarily data portability. Availability of additional binary integer sizes and alignments is important in a few applications.
Do these types have any use outside of I/O? Probably not, except for native endianness which can be used for fine grained control over side and alignment.
Is there is a performance hit when using these types? Yes, for sure, compared to arithmetic operations on native integer types. However, these types are usually be faster, and sometimes much faster, for I/O compared to stream inserters and extractors.
Which is better, big-endian or little-endian? Big-endian tends to be a bit more of an industry standard, but little-endian may be preferred for applications that run primarily on x86 (Intel/AMD) and other little-endian CPU's. The Wikipedia article gives more pros and cons.
What good is native endianness? It provides alignment and size guarantees not available from the built-in types. It eases generic programming.
Why bother with the aligned endian types? Aligned integer operations may be faster (20 times, in one measurement) if the endianness and alignment of the type matches the endianness and alignment requirements of the machine. On common CPU architectures, that optimization is only available for aligned types. That does allow I/O of maximally efficient types on an application's primary platform, yet produces data files are portable to all platforms, but the code is likely to be more fragile and less portable than with the unaligned types.
These types are really just byte-holders. Why provide the arithmetic operations at all? Providing a full set of operations reduces program clutter and makes code both easier to write and to read. Say you have to increment a variable in a record. It is very convenient to write:
++record.foo;
Rather than:
int temp( record.foo); ++temp; record.foo = temp;
The endian_example.cpp program writes a binary file containing four byte big-endian and little-endian integers:
#include <iostream> #include <cassert> #include <cstdio> #include <boost/integer/endian.hpp> using boost::integer::big32_t; using boost::integer::little32_t; namespace { // This is a portion of a widely used GIS file format. I have no idea why // anyone would mix big and little endians in the same format - but it is // a real format and users wishing to write code manipulating files in that // format have to deal with it. struct header { big32_t file_code; big32_t file_length; little32_t version; little32_t shape_type; }; const char * filename = "test.dat"; } int main() { assert( sizeof( header ) == 16 ); header h; h.file_code = 0x04030201; h.file_length = sizeof( header ); h.version = -1; h.shape_type = 0x04030201; // Low-level I/O such as POSIX read/write or <cstdio> fread/fwrite is // typically used for binary file operations. Such I/O is often performed in // some C++ wrapper class, but to drive home the point that endian integers // are usually used in fairly low-level code, <cstdio> fopen/fwrite is used // for I/O in this example. std::FILE * fi; if ( !(fi = std::fopen( filename, "wb" )) ) { std::cout << "could not open " << filename << '\n'; return 1; } if ( std::fwrite( &h, sizeof( header ), 1, fi ) != 1 ) { std::cout << "write failure for " << filename << '\n'; return 1; } std::fclose( fi ); std::cout << "created file " << filename << '\n'; return 0; }
After compiling and executing endian_example.cpp, a hex dump of test.dat
shows:
0403 0201 0000 0010 ffff ffff 0102 0304
Classes with similar functionality have been independently developed by several Boost programmers and used very successful in high-value, high-use applications for many years. These independently developed endian libraries often evolved from C libraries that were also widely used. Endian integers have proven widely useful across a wide range of computer architectures and applications.
Original design developed by Darin Adler based on classes developed by Mark
Borgerding. Four original class templates combined into a single endian
class template by Beman Dawes, who put the library together, provided
documentation, and added the typedefs. He also added the unrolled_byte_loops
sign partial specialization to correctly extend the sign when cover integer size
differs from endian representation size.
Comments and suggestions were received from Benaka Moorthi, Christopher Kohlhoff, Cliff Green, Gennaro Proto, Jeff Flinn, John Maddock, Kim Barrett, Marsh Ray, Martin Bonner, Matias Capeletto, Rene Rivera, Scott McMurray, Sebastian Redl, Tomas Puverle, and Yuval Ronen.
Last revised: 03 July, 2006
© Copyright Beman Dawes, 2006
Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at www.boost.org/ LICENSE_1_0.txt)