OTPCL

Open Telecom Platform Command Language

OTPCL Version 0.3.0: The Stringier The Better

It’s been a long while since the last release, but OTPCL ain’t dead quite yet!

This release involves two major changes to how OTPCL works:

  1. Support for “stringy” interpreters

  2. A completely different way to define OTPCL commands

What in tarnation is a “stringy” interpreter?

So by default, OTPCL’s parser is “smart” about types; if it parses an integer it’ll emit an integer, if it parses a float it’ll emit a float, and if it parses an atom it’ll emit an atom. This is great for a general-purpose(-ish) Erlang-compatible language, since it means that if you want to call, say, math:ceil/1, you could do use math; math ceil 1.2 in OTPCL and you’d get a sensible 2.0. This is not so great, however, in two situations:

  1. You want to use OTPCL to parse user-provided config files or scripts, and you’re worried about careless or outright malicious users uploading OTPCL configs/scripts that create lots of atoms and bring down the Erlang VM.

  2. You want to use OTPCL to parse config files and want to avoid variants of the Norway problem (namely: the “string or float?” example)

To address this, OTPCL 0.3.0 introduces a new “stringy” mode for the interpreter. Stringy interpreters, as the phrase might imply, don’t do any special parsing of “atomic” values (atoms, integers, or floats), instead emitting them as (binary) strings. This works through a special variable ($STRINGY_INTERPRETER); if that’s set, then the interpreter is stringy. For example, in an OTPCL script or REPL:

import erlang
is_atom foo    # returns true
set STRINGY_INTERPRETER ok
is_atom foo    # returns false
unset STRINGY_INTERPRETER
is_atom foo    # returns true again

There’s of course a helper function in the Erlang-facing API (otpcl_env:stringy_state/0) that will generate such a state for you. This starting state is based on otpcl_env:core_state/0, which only includes the commands defined in otpcl_core (that is: return and |) - meaning that scripts running with such a state won’t be able to break out of stringiness by calling set STRINGY_INTERPRETER ok, since they won’t have a set command at all by default.

The downside of stringiness is that there’s no automatic conversion at the moment when calling Erlang functions; all arguments passed are strings, so any function such an interpreter calls must be prepared for that, or else you’re in for a bad time:

otpcl> set STRINGY_INTERPRETER ok
ok
otpcl> math ceil 1.2
error: badarg
Stacktrace:
  math:ceil/[<<"1.2">>]
    error_info: #{module => erl_stdlib_errors}
  otpcl_meta:'-fun2cmd/2-fun-0-'/4
    file: "/home/northrup/Projects/otpcl/src/otpcl_meta.erl"
    line: 212
  otpcl_eval:interpret/2
    file: "/home/northrup/Projects/otpcl/src/otpcl_eval.erl"
    line: 223
  otpcl_repl:eval/2
    file: "/home/northrup/Projects/otpcl/src/otpcl_repl.erl"
    line: 56

Also, since stringy_state/1 is just core_state/1 with $STRINGY_INTERPRETER set to ok, it’s up to you to build up (with e.g. otpcl_meta:import/2,3 or otpcl_meta:use/2,3) an environment in which scripts run.

The upside, however, is that - as long as you use otpcl_env:stringy_state/1 as your starting state and take care to avoid importing/useing any commands/functions that are able to break out of stringiness or dynamically create atoms - you should be safe to interpret even adversarial scripts, and you can rest easy knowing that OTPCL won’t do any type-related funny business. Going forward, I’m making it a point to maximize the number of OTPCL commands in the standard library (or what passes for one, at least) that work with stringy interpreters - that is, being lenient on whether passed-in arguments are strings or atoms/integers/floats (and converting as necessary, with appropriate warnings in the docs if said conversion entails dynamic atom creation or anything else that might make adversarial inputs unsafe to interpret).

Defining OTPCL commands works differently now?

In OTPCL 0.2.0 (and earlier), if you wanted to create an Erlang module that exported OTPCL commands, you would do something like this:

-module(foo).
-export([bar/2]).
-otpcl_cmds([bar]).

bar([], State) ->
    {baz, State}.

And then we’d be able to call it from OTPCL:

use foo
foo bar  # returns baz
# Alternately...
import foo
bar      # also returns baz

That -otpcl_cmds([bar]). is a bit cumbersome, though, since it means having to maintain two separate lists of exports (one for Erlang, one for OTPCL). That’s more things to think about and more things to mess up.

This is even worse if you’re doing this in Elixir, since Elixir doesn’t persist module attributes by default; you have to register that otpcl_cmds attribute and specify that it should be persisted, which means yet more boilerplate:

defmodule Foo do
  Module.register_attribute(__MODULE__, :otpcl_cmds, persist: true)
  @otpcl_cmds [:bar]

  def bar([], state), do: {:baz, state}
end

…yuck!

OTPCL 0.3.0 overhauls this a fair bit by instead using prefixes on function names to denote whether something is an OTPCL command. The 0.3.0 version of the above Erlang module:

-module(foo).
-export('CMD_bar'/2).

'CMD_bar'([], State) ->
    {baz, State}.

And in Elixir:

defmodule Foo do
  def unquote(:"CMD_bar")([], state), do: {:baz, state}
end

…okay so for the Elixir version we trade some ugliness for some different (but IMO less severe) ugliness, but the Erlang version is much nicer, no? This approach has quite a bit of precedent, too; EUnit, for example, uses the _test suffix to know which exported functions in a module are tests, and Elixir macros are actually ordinary Erlang functions with a MACRO- prefix.

But wait, there’s more! If OTPCL sees an exported function without a prefix, it’ll turn that into a command, too - specifically, a “pure” command, i.e. one which doesn’t alter state. We could rewrite that Erlang module like so:

-module(foo).
-export(bar/0).

bar() ->
    baz.

Or that Elixir module like so:

defmodule Foo do
  def bar(), do: :baz
end

And those will still work exactly the same way from OTPCL’s perspective:

use foo; foo bar  # still returns baz
import foo; bar   # also still returns baz

All this because OTPCL’s use and import commands are now smart enough to differentiate between “hey, this 2-arity function has a CMD_ prefix, so this is probably an OTPCL command and I should have the interpreter pass its state into its second argument and expect to get back a new state in the returned tuple’s second member” v. “hey, this function doesn’t have a prefix so it’s probably an ordinary pure function and therefore the interpreter can just call it and set the result as $RETVAL”, and that smartness is all thanks to this new method of defining OTPCL commands in Erlang modules.

But wait, there’s more! Because of this new approach of prefixing commands instead of separately listing them in a custom attribute, it’s now possible to expose Erlang and OTPCL APIs for the same module. Before, that required either exposing OTPCL-isms to your module’s users or else doing a bunch of funky conversions to/from OTPCL’s list + state calling convention; now, you just define some_function/N and 'CMD_some_function'/2 and you’re all set. OTPCL’s own code heavily exploits this to make as much of OTPCL as usable as possible from both the host language and OTPCL itself.

This is (obviously) a breaking change, though. I did not retain any code to support the old way of defining OTPCL-command-exporting modules, so if you’ve been using OTPCL already (despite it being nowhere near production ready) you’ll need to migrate over to the new approach.

Other stuff

Test coverage is vastly improved now. Having comprehensive tests was kinda mandatory for such a large overhaul, and it helped with tracking what was converted to the prefixed command approach v. what was still on the module attribute approach.

I also removed otpcl_shell for the time being. I figure in the short term it’s easy enough to pull in the Erlang functions for which that module’s functions were thin wrappers around (and in the long term, an otpcl_shell module that actually made OTPCL suitable as a system shell replacement would look very different from such a module of thin wrappers). Besides, having to write tests for something as finicky to test as operating system interactions struck me as not worth the effort at the moment.