Mircea Baja @ Sophos - 15th May 2018

Boilerplate, metaprogramming, reflection in C++

Repeat of the presentation I gave @ ACCU Oxford - 29th January 2018

Executive summaryReflective metaprogramming = generate code based on code
We need better reflective metaprogramming facilities in C++
There are proposals to address it
It's a complex issue
Standardisation will likely happen after 2020
2

Sample problem3

Data

{
  "name": "Kipper",
  "breed": "Labrador"
}

Traditional bad C++ solution

class bad_dog
{
private:
  std::shared_ptr<std::string> name_;
  std::shared_ptr<std::string> breed_;
public:
  std::shared_ptr<std::string> get_name();
  void set_name(std::shared_ptr<std::string> value);
  std::shared_ptr<std::string> get_breed();
  void set_breed(std::shared_ptr<std::string> value);
  void init(const Json & doc);
};

Issues with this traditional solutionReference semantics leading to
Difficulty of local reasoning and
Complex memory layout leading to
Degraded performance
Getter and setters leading to
Combinatoric interface
Init method leading to
Multi-step initialization
Multiple responsibilities leading to
Maintainability and
Testing difficulties
7

Bad memory layout

Better C++ solution

struct dog
{
  std::string name;
  std::string breed;
};
dog dog_from_json(const Json & doc);

Better C++ solution

struct dog
{
  std::string name;
  std::string breed;
};
dog dog_from_json(const Json & doc);

Physical layout:

dog struct is defined in dog.h
Serialization in dog_from_json.h and cpp

Memory layout

Boilerplate12

Deserialization

dog dog_from_json(const Json & doc)
{
  dog x;
  x.name = doc['name'].as_string();
  x.breed = doc['breed'].as_string();
  return x;
}

Deserialization

dog dog_from_json(const Json & doc)
{
  dog x;
  x.name = doc['name'].as_string();
  x.breed = doc['breed'].as_string();
  return x;
}

dog dog_from_json(const Json & doc)
{
  return {
    doc['name'].as_string(),
    doc['breed'].as_string()
  };
}

Equality

bool operator==(const dog & left, const dog & right) noexcept
{
  if (left.name != right.name)
  {
    return false;
  }
  return left.breed == right.breed;
}

Equality

bool operator==(const dog & left, const dog & right) noexcept
{
  if (left.name != right.name)
  {
    return false;
  }
  return left.breed == right.breed;
}

bool operator==(const dog & left, const dog & right) noexcept
{
  return (left.name == right.name) &&
    (left.breed == right.breed);
}

Order

bool operator<(const dog & left, const dog & right) noexcept
{
  if (left.name < right.name)
  {
    return true;
  }
  if (left.name > right.name)
  {
    return false;
  }
  return left.breed < right.breed;
}

Order

bool operator<(const dog & left, const dog & right) noexcept
{
  if (left.name < right.name)
  {
    return true;
  }
  if (left.name > right.name)
  {
    return false;
  }
  return left.breed < right.breed;
}

bool operator<(const dog & left, const dog & right) noexcept
{
  if (left.name != right.name)
  {
    return left.name < right.name;
  }
  return left.breed < right.breed;
}

Other comparison operators

bool operator!=(const dog & left, const dog & right) noexcept
{
  return !(left == right);
}
bool operator<=(const dog & left, const dog & right) noexcept
{
  return !(right < left);
}
bool operator>(const dog & left, const dog & right) noexcept
{
  return right < left;
}
bool operator>=(const dog & left, const dog & right) noexcept
{
  return !(left < right);
}

NoteImplementation on previous slide is not optimal (reason: unnecessary multiple traversal)
Spaceship operator '<=>' proposal
Note that equality and comparison do not have the same cost
Pure historical reasons for the compiler not implementing comparison operators by default
20

Padding

Metaprogramming currently22

Tie members alternative

auto tie_members(const dog & x) noexcept
{
  return std::tie(x.name, x.breed);
}
// returns a std::tuple<const std::string &, const std::string &>

Tie members alternative

auto tie_members(const dog & x) noexcept
{
  return std::tie(x.name, x.breed);
}
// returns a std::tuple<const std::string &, const std::string &>

bool operator==(const dog & left, const dog & right) noexcept
{
  return tie_members(left) == tie_members(right);
}
bool operator<(const dog & left, const dog & right) noexcept
{
  return tie_members(left) < tie_members(right);
}
// etc.

Tie with (some) check

template<class T, typename ... Args>
auto tie_with_check(Args & ... args) noexcept
{
  static_assert(sizeof(T) == sizeof(std::tuple<Args...>),
    "You forgot a member variable");
  return std::tie(args...);
}

Tie with (some) check

template<class T, typename ... Args>
auto tie_with_check(Args & ... args) noexcept
{
  static_assert(sizeof(T) == sizeof(std::tuple<Args...>),
    "You forgot a member variable");
  return std::tie(args...);
}

auto tie_members(const dog & x) noexcept
{
  return tie_with_check<dog>(x.name, x.breed);
}

Tie with (some) check

template<class T, typename ... Args>
auto tie_with_check(Args & ... args) noexcept
{
  static_assert(sizeof(T) == sizeof(std::tuple<Args...>),
    "You forgot a member variable");
  return std::tie(args...);
}

auto tie_members(const dog & x) noexcept
{
  return tie_with_check<dog>(x.name, x.breed);
}

'tuple' is usually implemented using recursive derivation, not as side by side member declaration as a struct.

Therefore there is no guarantee that std::tuple has the same layout as the struct.

Padding might be different.

Tuple inheritance

Struct layout

template<typename ... Args>
struct struct_layout;
template<typename T0>
struct struct_layout<T0>
{
  T0 m0;
};
template<typename T0, typename T1>
struct struct_layout<T0, T1>
{
  T0 m0;
  T1 m1;
};
// and so on up to the max number of members you support

Struct layout

template<typename ... Args>
struct struct_layout;
template<typename T0>
struct struct_layout<T0>
{
  T0 m0;
};
template<typename T0, typename T1>
struct struct_layout<T0, T1>
{
  T0 m0;
  T1 m1;
};
// and so on up to the max number of members you support

  static_assert(sizeof(T) == sizeof(struct_layout<Args...>),
    "You forgot a member variable");

Issues with current metaprogramming in C++It looks clever, but it is convoluted
Conditionals ('if') use overloading/template specialization
Iterations ('for') use recursion (e.g. recursive inheritance)
Store state as parametrized types and constants
Instantiate template to trigger the computation
Relies on the historical accident that the templates machinery is Turing
complete
31

Issues with current reflection support in C++Can query for types e.g. std::is_pointer
Can do simple type transformations e.g. std::make_unsigned
But is very limited, relies on overloading, template specialization, concepts
Can't enumerate members
Can't get names
32

Other metaprogramming optionsPre-processor
constexpr functions
Other languages/tools outside C++ (e.g. MIDL compiler etc.)
33

Use cases for better facilities34

Use case: common operators==, < implemented lexicographically
implement !=, <=, >=, >  in terms of the above
std::hash
ProblemsCan be do all in one go or piecemeal?
Some functions are, some function are not members of the class
std::hash might be in a different namespace (the need to specify where code
generation takes place)
35

Use case: serializationto/from JSON/XML
relational database mapping
logging function arguments
ASSERT_EQ macro in tests
ProblemsGetting the current function as input for reflection
Getting names of the parent context
Getting source code line of the original construct
36

Use case: enum to/from string

enum class foo_bar
{
  foo,
  bar
};
auto to_string(foo_bar x) noexcept
{
  switch (x)
  {
  case foo_bar::foo:
    return "foo";
  case foo_bar::bar:
    return "bar";
  default:
    std::terminate();
  }
}

Use case: transformationC structure mapping a C++ structure members
mock class for a interface
member functions wrapping (e.g. with logging, locking, Pimpl)
stand alone function wrapping (e.g. variations of error handling)
ProblemsWhich file (even language) is the code generated in
Dealing with templates (class or function)
38

Use case: identifiers on the fly

currently they require macros
low hanging fruit from the code generation component of the problem

Theory40

Reflection workflow

reflect on a source construct
analyse
generate something to execute

the steps above for equality or from_json:

get a reflection of the class
get a reflection of each member
maybe check the type of the member
generate a code snippet

Static vs dynamic reflection

A lot of languages support dynamic reflection:

reflection/generation not at compile time
not a zero-cost abstraction, it forbids optimisations of original source code constructs.
generation could mean "function invocation" at runtime

For C++ we want static reflection metaprogramming

Static reflection

based on a source construct
the reflection operator returns a description of the source construct
code generation is based on a description of the source construct
it all happens at compile time

Domains and operators

Input domain

Yes:

class
- data members
- member functions
- member types
enumerators: e.g. serialization for enums
functions (overloads?)

Maybe:

variables
class templates and function templates
attributes

Probably not:

expressions

Reflection result

Metadata:

name (e.g. name of a class)
list of members
arguments of a function
etc.

Options:

unique type for each reflection
objects, but still of unique type for each reflection
objects of a single type e.g. std::meta::info

Code generation optionsexisting template metaprogramming
raw string (powerful, but could be opaque)
abstract syntax tree creation API
token sequence (e.g. -> { ...code here...})
associate a name as a shorthand for code generation (e.g. metaclasses)
47

Proposals48

Proposals stack

Upstairs:

P0707 Metaclasses by Herb Sutter

Downstairs:

P0194 by Matus Chochlik, Axel Naumann, David Sankel
P0590 by Andrew Sutton, Herb Sutter

Plumbing:

A lot of plumbing required: e.g. constexpr_vector, heterogeneous for loop

Downstairs50

P0194, P0385

by Matus Chochlik, Axel Naumann, David Sankel

Type-based reflection with template metaprogramming

Uses 'reflexpr' as the reflection operator
Yields a new type for each reflection
Properties are static members of the class
Code generation operators are type traits or 'unreflexpr' for more complex cases

Advantages:

Powerful
Few additions to language, few compiler intrinsics

Disadvantages:

But has downside of using templates machinery to perform computation (complex style compared with plain C++ programming)
A lot of classes generated: low compile time performance

P0385 sample (part 1)

template<typename T>
struct compare_data_members
{
  ...
};
template<typename T>
bool generic_equal(const T & a, const T & b)
{
  using metaT = $reflect(T);
  bool result = true;
  reflect::for_each<reflect::get_data_members_t<metaT>>(
    compare_data_members<T>{a, b, result});
  return result;
}

P0385 sample (part 2)

template<typename T>
struct compare_data_members
{
  const T & a;
  const T & b;
  bool & result;
  template<reflect::Object MetaDataMember>
  void operator ()(MetaDataMember) const
  {
    auto mem_ptr = reflect::get_pointer_v<MetaDataMember>;
    result &= a.*mem_ptr == b.*mem_ptr;
  }
};

P0590, P0589, P0712

by Andrew Sutton, Herb Sutter

Type-based reflection with heterogeneous containers

Uses $ as the reflection operator
$ applies to: variables, types, namespaces, functions
The return type depends on what was reflected on (it's a template specialization, but you would use auto all the time anyway)
Properties are static constexpr members
Code generation operators as 'typename', 'namespace', 'idexpr'
Requires more compiler intrinsics
Because they are template specialization properties are not instantiated until called (sometimes good, sometimes bad)
Getting class members returns a heterogeneous tuple-like container

Disadvantages:

Still generates a lot of types: resource intensive compiler and complex usage style

Plumbing55

Reflection operator $

void foo(int n) {
  int x;
  auto r1 = $int; // r1 has type meta::fundamental_type<X>
  auto r2 = $foo; // r2 has type meta::function<X>
  auto r3 = $n; // r3 has type meta::parameter<X>
  auto r4 = $x; // r4 has type meta::variable<X>
}

Heterogeneous 'for' loop

Aka. unrolled loop, loop expansion, tuple-based for loop

for... (auto x : $s.member_variables())
{
  std::cout << x.name();
}

Heterogeneous 'for' loop

Aka. unrolled loop, loop expansion, tuple-based for loop

for... (auto x : $s.member_variables())
{
  std::cout << x.name();
}

Expands to

auto && tup = $s.member_variables();
{ auto x = std::get<0>(tup); cout << x.name(); }
{ auto x = std::get<1>(tup); cout << x.name(); }

Existing constexpr

constexpr int one_bigger(int x)
{
  return ++x;
}
int some_array[one_bigger(6)];
int main(int /*argc*/, char * argv[])
{
  std::cout << std::size(some_array) << '\n';
  int argument = std::stoi(argv[1]);
  int result = one_bigger(argument);
  std::cout << result << '\n';
}

constexpr vs immediateconstexpr means 'it can be used at compile time'
constexpr does not mean 'it will be used at compile time'
60

constexpr blocks

constexpr
{
  // do some compile time programming here
}

Can be placed in namespace, class, block scope.

constexpr blocks

constexpr
{
  // do some compile time programming here
}

Can be placed in namespace, class, block scope.

Equivalent to:

constexpr void __unnamed_fn()
{
  // compile time code here
}
constexpr int __unnamed_var = (__unnamed_fn(), 0);

Injection statement

Can appear inside a constexpr block

constexpr
{
  -> namespace { int f() { return 0; } }
} // injection site is here

Injection statement

Can appear inside a constexpr block

constexpr
{
  -> namespace { int f() { return 0; } }
} // injection site is here

Equivalent to:

int f() { return 0; }

Injection statement

Can appear inside a constexpr block

constexpr
{
  -> namespace { int f() { return 0; } }
} // injection site is here

Equivalent to:

int f() { return 0; }

Several options for the scope of the injection: namespace, class, block, expression.

All together now

template<Enum E>
const char * to_string(E value)
{
  switch(value)
  constexpr {
    for... (auto e : $E.enumerators())
    {
      -> { case e.value(): return e.name(); }
    }
  }
}

Identifier generation

void foo() { ...  }
void foo_bar() { ... }
void g()
{
  auto x = $foo;
  return declname(x "_bar")();
}

Other plumbingstd::const_vector
Handling strings at compile time
Custom error messages (diagnostic)
Print generated constructs (diagnostic)
68

Named constexpr blocks

constexpr void make_links()
{
  -> class C
  {
    C * next;
    C * prev;
  }
}
struct list
{
  constexpr { make_links(); }
};

Named constexpr blocks

constexpr void make_links()
{
  -> class C
  {
    C * next;
    C * prev;
  }
}
struct list
{
  constexpr { make_links(); }
};

Equivalent to:

struct list
{
  list * next;
  list * prev;
};

Upstairs71

Metaclass sample (part 1)

$class interface
{
  constexpr {
    compiler.require($interface.variables().empty(),
      "interfaces may not contain data");
    for (auto f : $interface.functions())
    {
      compiler.require(!f.is_copy() && !f.is_move(),
         "interfaces may not copy or move; consider a "
         " virtual clone() instead");
      if (!f.has_access()) f.make_public();
      compiler.require(f.is_public(),
        "interface functions must be public");
      f.make_pure_virtual();
    }
  }
  virtual ~interface() noexcept { }
};

Metaclass sample (part 2)

interface Shape
{
  int area() const;
  void scale_by(double factor);
};

Alternatives: Source to destination classchange in place
two types
two types (source is hidden)
74

.is and .as

struct legacy_point { int x; int y; }; // in C++17 this is not comparable...
set<legacy_point> s; // and so this is an error
using ordered_point = $legacy_point.as(ordered); // ... but this is ordered
set<ordered_point> s; // so this is OK

P0707 Metaclasses

by Herb Sutter

Advantages:

ability to create constructs that currently require new language features like 'enum class'
$T.as(M) is very powerful
composes (e.g. using inheritance syntax

Maybe disadvantage:

a bit too class oriented (e.g. friend trick)

Metaclass sample (part 3)

$class ordered
{
  constexpr {
    if (! requires(ordered a) { a == a; })
    -> {
      friend bool operator == (const ordered& a, const ordered& b) {
        constexpr {
          for (auto o : ordered.variables())
          -> { if (!(a.o.name$ == b.(o.name)$)) return false; }
        }
        return true;
      }
    }
  //...
  }
};

Metaclass sample (part 4)

ordered dog
{
  std::string name;
  std::string breed;
};

Observations79

Relation to template metaprogrammingReflective metaprogramming has the potential to replace most/all usage of
template metaprogramming (but not clear to what degree)
80

Relation to D lang introspectionWhat Andrei Alexandrescu called the need for compile time introspection
facilities, criticising the concepts spec for C++
Similarly Andrei Alexandrescu demonstrates the ability to develop new
features like bit-fields using existing language constructs rather than
having to add new language extensions (with metaclasses in particular)
81

Similarities to template instantiationEspecially when we get to code generation at what time the metadata is
captured, at what point evaluated?
Entity is not yet complete inside member function of a class
Or inside of a function ... what is the return type if auto?
Or inside a template function
82

Famous last wordsIt can be misused
But it can be used for good as well:  high return feature to eliminate boilerplate and most template programming
Tooling improvements required
83

References - 1

Andrew Sutton
CppCon 2017: Reflection
https://www.youtube.com/watch?v=N2G-Frv1z5Q

Andrew Sutton
CppCon 2017: Meta
https://www.youtube.com/watch?v=29IqPeKL_QY

Herb Sutter
CppCon 2017: Meta: Thoughts on generative C++
https://www.youtube.com/watch?v=4AfRAVcThyA

Herb Sutter
Thoughts on Metaclasses - Keynote [ACCU 2017]
https://www.youtube.com/watch?v=6nsyX37nsRs

Matúš Chochlík, Axel Naumann, David Sankel
Static Reflection in a Nutshell
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0578r1.html

References - 2

Matúš Chochlík, Axel Naumann, David Sankel
Static reflection - Rationale, design and evolution
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0385r2.pdf

Matúš Chochlík, Axel Naumann, David Sankel
Static reflection
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0194r4.html

Matúš Chochlík, Axel Naumann, David Sankel
Static reflection of functions
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0670r1.html

Andrew Sutton, Herb Sutter
A design for static reflection
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0590r0.pdf

Andrew Sutton
Tuple-based for loops
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0589r0.pdf

References - 3

Andrew Sutton, Herb Sutter
Implementing language support for compile-time metaprogramming
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0712r0.pdf

Herb Sutter
Metaclasses : Generative C++
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0707r2.pdf

Daveed Vandevoorde, Louis Dionne
Exploring the design space of metaprogramming and reflection
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0633r0.pdf

Daveed Vandevoorde
Reflect Through Values Instead of Types
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0598r0.html

Daveed Vandevoorde
Reflective Metaprogramming in C++ 2003
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1471.pdf

References - 4

Mike Spertus
Use Cases for Compile-Time Reflection
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3492.pdf

Jeff Snyder, Chandler Carruth
Call for Compile-Time Reflection Proposals
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3814.html

Daveed Vandevoorde
std::constexpr_vector
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0597r0.html

Andrei Alexandrescu
Fastware - ACCU 2016
https://www.youtube.com/watch?v=AxnotgLql0k

Labrador picture
https://pixabay.com/en/animal-dog-puppy-pet-photography-2184791/

Thanks

Thanks go to Nigel Lester for organizing the original ACCU talk and spell checking the presentation.

Questions?

https://bajamircea.github.io/presentations/2018-05-15-reflection.html

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help