Blog posts on Tamás K. Papp's website

Julia and batteries

2019-12-21T16:33:38+01:00

New Julia users frequently suggest that some features should be included in Base (the part that is available directly without using any packages) or the standard libraries. The colloquial phrase is that “Julia should come with batteries included”, hence the title of this post.

In this blog post, I explain why doing this is unlikely to improve anything, likely to make some things worse, and thus often meets with resistance from developers. Most of these points are well-known, and repeated in various discussions. I thought it would be nice to have them in one place.

About the standard libraries

Before the 0.7/1.0 release of Julia, a lot of functionality now in the standard libraries was part of Base. As the codebase grew, this was causing practical problems, so modularizing the code was on the agenda from the very beginning, with the understanding that this would make things load faster, encapsulate implementation details better, and the code would be easier to maintain. The excision was completed for 0.7, and “standard libraries” as we know them now were born.

One very neat thing about Julia is that conceptually, standard libraries are like any other package. There is no “secret sauce” that makes them special: some things like precompilation are enabled by default, but you can do the same for any package.

What is different about standard libraries is an implicit guarantee for code quality and support: they are officially endorsed, and the Julia maintainers fix issues relating to these packages.

When users suggest that other packages they find useful are included in the standard libraries, they usually expect that the same kind of code quality and support guarantees would extend automatically to these packages.

Unfortunately, since developer time is a scarce resource, this would only mean that the same number of people would now have to maintain more code. This would most likely result in longer delays for releases, and is not necessarily a guarantee of better support: some standard libraries have (minor) issues that have been outstanding for a long time.

The benefits of (third party) packages

Below, I list some major advantages of having code in third party packages (what we normally think of as “packages”, usually registered) that are maintained outside the Julia repository.

Separate versioning, more frequent releases

Standard libraries, not being versioned separately, maintain the same SemVer compatibility guarantees as the rest of Julia. This means that no incompatible changes are allowed without changing the major version — in other words, anything that ends up there would have to be supported without breaking changes for the rest of the 1.x life cycle.

Since nontrivial interfaces usually require a few iterations to become polished, this would mean that the authors would need to get it right without a lot of community testing, or keep legacy methods around for the rest of 1.x.

Third party packages have their own version. This means that they can release as often they like, experiment with new ways of doing things and deprecate old APIs, and add and reorganize features without the implicit cost of having to support them for a long time. Because of this, it is not uncommon for actively maintained packages to have minor releases with new features every month or so, and bugfix releases within a day.

Easier contribution

Contributing to standard libraries is usually a more involved process than making a PR to third party packages. Unit tests take longer to run (since the whole of Julia is tested), ramifications of changes have to be considered in wider context, and contributors have to be much more cautious with experimentation since anything that ends up there would need to be supported for a long time.

The changes will need to be reviewed by people who may be very busy working on some other part of Julia, and may take a longer time; decisions frequently need a wider consensus and some PRs just languish unresolved for months. This naturally raises the barrier to entry, and can be quite frustrating to contributors.

In contrast, many packages have a single or a few authors, who have a clear vision of where they want to take their package, and are usually able to make decisions quicker.

Separate documentation and issue tracker

The Julia repository has a total of 17k issues (!), 14k of them closed, a similar number of open pull requests, and a manual that has 400 pages on the standard libraries (in PDF format). It is occasionally challenging to find an earlier issue, discussion, or documentation for something.

For third party packages, having a separate issue tracker and documentation makes it much easier to find what you are looking for.

Development happens in packages

If you would like Julia to support something that is experimental or not fully specified (and most software isn’t), it is unlikely that adding it to Base or the standard libraries is the right solution.

Instead, it is usually recommended that you put it in a package and work out the details, improving the code based on experience and feedback. There is no downside to this in most cases, just the benefits mentioned above.

If the experiment pans out, great: you have created a nice package! If it doesn’t, it can be retired and archived, without having to commit to continued support.

Comments reenabled

2019-10-10T17:40:39+02:00

I have re-enabled comments using utterances, which uses Github issues to record and render comments. You need to login via Github to comment.

How my Julia coding style changed

2019-09-07T08:01:57+02:00

The recent redesign of DynamicHMC.jl provides an opportunity to reflect on how my Julia coding style changed. I started this package two years ago, and did not do any major refactoring after it became usable, mostly because I did not have a good comprehensive test suite or a solid conceptual understanding of all the ramifications of the algorithm, especially the adaptation, and I was afraid of breaking something that I just got working. Consequently, the code before the redesign largely reflected the coding style I developed for Julia 0.5 and then partly 0.6.

I am one of those people who recognize the benefits of reflecting on the past from time to time, but are too ~~busy~~ lazy to keep a diary. But code under version control is effectively a diary about the life of software, and I find it interesting to think about how my Julia coding style has changed in the past to years.

It is important to emphasize that this writeup reflects how I code at the moment. I recognize that other coding styles in Julia can be perfectly legitimate, and when I contribute to packages, I do my best to follow existing style. Also, it is very likely that my own coding style will change as the language evolves. In this blog post, I won't address details of coding style about which there is a more or less general agreement --- see the Style Guide in the manual and YASGuide for useful summaries. Finally, some examples are stylized adaptations of real code for simpler exposition.

Randomness

Random numbers are very important for scientific computing, but at the same time present an important challenge for writing bug-free parallel code and unit testing. A lot of code in Random and Base follows the interface conventions

rand([rng], ...)

making the random number generator an optional first argument, and defaulting to the global one.

While this can be convenient for user code, in packages I find that it is better to keep the random number generator as an explicit argument to almost all methods. This paves the way for making the code work nicely with composable multi-threading too.

If random and deterministic parts of the algorithm can be separated, it should be done, as it usually makes the code cleaner, and unit testing much easier. An example of this is factoring out the random directions for tree doubling in NUTS: drawing a single set of bit flags once and providing this to the tree building algorithm, the latter can treat the doubling directions as predetermined. This also made it much easier to verify detailed balance in the unit tests.

Named tuples

Named tuples are extremely useful for organizing multiple values as inputs and outputs when a composite type (struct) is not warranted or needed. If a function returns multiple values, especially more than two, I now always default to using named tuples, ie

function f(x, y)
    a = ...
    b = ...
    c = ...
    (a = a, b = b, c = c)
end

then use @unpack from Parameters.jl for destructuring:

@unpack a, b = f(x, y) # I don't need c here

This is costless and makes the interfaces more flexible: for example, I can add extra fields, or eventually make the returned value a composite type later, without changing anything in the callers.

Multi-liners as opposed to one-liners

Almost all the “clever” one liners are now expunged from the code. Compare

EBFMI(x) = (πs = map(x -> x.π, x); mean(abs2, diff(πs)) / var(πs))

with

function EBFMI(tree_statistics)
    πs = map(x -> x.π, tree_statistics)
    mean(abs2, diff(πs)) / var(πs)
end

The first one is kind of crammed, and difficult to read, while the second one makes it clear that an interim value is obtained for the calculation.

When a function returns multiple values, I find it nice to assign each to a variable first, ie

function f(x, y)
    a = g(x, y)
    b = h(x, y)
    a, b
end

as opposed to

f(x, y) = g(x, y), h(x, y)

This makes the returned structure much more readable when I need to look at some code later.

That said, I still have a strong preference for writing small functions that “do one thing”. This again is frequently an optimal strategy for Julia, as the compiler is very good about inlining decisions (and when not, can be nudged with @inline).

Abstract types can be an implementation detail

Abstract types can be very useful for organizing method dispatch: code like

abstract type SomeAbstractType end

f(x::SomeAbstractType) = do_something()

struct ThatSubType{T} <: SomeAbstractType
    y::T
end

g(x::ThatSubType) = do_something_else(x.y)

can help avoid repetition.

But abstract types are just one convenient tool for this, and building a type tree is a tool, not an end in itself. When they are no longer adequate, other solutions such as traits should be used instead.

Consequently, these days I mostly consider type hierarchies an implementation detail and I am wary of exposing them in the API of a package. This is especially important if some functionality requires that the user of a package implements their own custom types, e.g. for a model. Requiring that

UserDefinedModel <: SomePackage.ModelType

to work with some API is usually unnecessary, and can cause unnecessary complications when the user would want UserDefinedModel to implement some other API. Either duck typing, or an explicit interface check like

implements_model(::Any) = false
implements_model(::Type{<:UserDefinedModel}) = true

can work better (returning types when that is necessary, eg for further dispatch — see Base.IteratorSize for an example).

Small type unions

The ability of the compiler create efficient code for small type unions made code organization much easier for me. For example, DynamicHMC.adjacent_tree attempts to build a tree next to some position, but this can fail because of divergence or turning in the leapfrog integrator, in which case it is useful to bail out early and return information about the reason. Consequently, this function either returns a tree, or an InvalidTree, which is a container with the aforementioned diagnostic information. The compiler recognizes this and generates fast code.

Some attention should be paid to avoid combinatoric type explosions, but when used well, small type unions can make code much simpler at little or no extra cost.

Functional is often the best solution

Preallocating containers for outputs can make code faster, but at the cost of extra complexity, especially that of having to figure out type parameters for said containers.

My general strategy is to write code in a functional, side-effect-free style until I am really convinced that preallocation would help, and then confine it to carefully compartmentalized parts of the code. This is of course ideal only for some kinds of code — when allocations dominate, a functional style is not optimal.

That said, Julia's compiler is getting more and more clever about optimizing allocations, so I frequently find that I never rewrite code with preallocated containers, and occasionally even revert from preallocated code to a functional style without a relevant performance cost.

When in doubt, document

Some readers of the code of DynamicHMC.jl will have the impression that it is overdocumented. Probably 20–25% of the lines in the source are docstrings, and most internal interfaces and convenience functions have one.

Primarily, this is driven by instances of the embarassing realization of having written some code two years ago which kind of works, but no longer being sure of all the details that would be nice to know before refactoring. But another consideration is encouraging contributions, or uses of the packages for research — besides being an implementation of the NUTS sampler, DynamicHMC.jl is also a platform for experimenting, which is easier to do when the internals are documented.

Julia's documentation ecosystem is now very mature and robust. Besides the excellent Documenter.jl, I also find DocStringExtensions.jl very useful.

Internal interfaces have high return on investment

Many packages, especially those that are work in progress or experimental, do not document their user interfaces very carefully. This is of course understandable, but I have developed a preference for abstracting out and documenting interfaces in internally used code, too, as the package matures.

This makes refactoring and compartmentalization much easier, and makes sense even if a particular interface has only a single implementation in the package, because unit tests can just provide a “dummy” implementation and test related code that way. An example of this is the code for “trees” and the related tests: trees here are an abstraction for walking integer positions in two directions. This is key for the NUTS algorithm, but can be handled as an abstraction separate from the details of leapfrog integrators.

ANN: DynamicHMC 2.0

2019-09-03T09:03:50+02:00

I am very pleased to announce version 2.0 of DynamicHMC.jl. I briefly summarize the new the developments in this blog post.

Code used in this post was written for Julia v1.* and, when applicable, reflects the state of packages used on 2019-09-03. You may need to modify it for different versions.

Note: my blog no longer has a comment section. Feel free to ask questions about this post on the Julia Discourse forum.

Some context

DynamicHMC.jl is part of a collection of packages for implementing Bayesian inference using a modular approach:

TransformVariables.jl just maps $\mathbb{R}^n$ vectors into collections of constrained parameters, like positive real numbers or valid correlation matrices,
LogDensityProblems.jl provides an interface for log density functions $\mathbb{R}^n \to \mathbb{R}$ and their derivative, also calculating the latter using automatic differentiation via one of the supported native Julia AD frameworks (see below),
DynamicHMC.jl, which just takes a log density and its gradient on $\mathbb{R}^n$, and performs MCMC using NUTS,
MCMCDiagnostics.jl for generic convergence diagnostics (ie not specific to a particular MCMC method).

In contrast to a single interface and a DSL for describing models, these packages provide a suite of tools for modern MCMC, with easily interchangable and modular building blocks. For example, the gradients of a log density can be obtained with

ForwardDiff.jl, which is very robust,
ReverseDiff.jl, which works better for medium-sized problems using a classic taped approach,
Flux.jl which is more modern and can be faster on larger problems, and of course
Zygote.jl which is very promising, but work in progress;

and switching between these usually just requires changing a single line. I find this especially important since as Zygote keeps maturing, it will play a more and more important role for fast AD. Moreover, the user is free to code all gradients manually, or just parts of them to help out AD, for example with the adjoint method.

In addition to this, some packages use DynamicHMC.jl as a backend. This is encouraged and remains to be supported. Finally, the third common use case for this package is as a research platform for experimenting with modern HMC algorithms: this is supported by a detailed documentation of internals.

Why a new API was needed

As bug reports and test cases kept accumulating, it was very clear that better adaptation heuristics, and a more sophisticated diagnosic and warmup interface was needed. When NUTS/HMC goes wrong with models it should be able to handle otherwise, the problem is usually with adaptation. The user should be able to learn what went wrong, and either manually tune stepsize and the kinetic energy metric, or choose an adaptation better suited to the model.

The old API of DynamicHMC was lacking in several ways. The main entry point was something like

chain, tuning = NUTS_init_tune_mcmc(∇P, 1000)

with tuning containing the adapted stepsize and kinetic energy metric.

In practice, it turns out that

either the model works fine, and then the user cares little about warmup,
or it doesn't, then information about the warmup would be necessary to debug the model.

Fine-tuning the adaptation sequence was possible, but rather unintuitive in the old API. I doubt that many people used this feature, or were even aware that it was available. Statistics on why adaptation failed or sampling didn't mix were also difficult to obtain.

After collecting the various problems, I spent some time on redesigning the internals, then the API to address all major issues.

Here I would like to thank all users of this library who provided a lot of valuable feedback, especially in the form of bug reports which I could study and use to tweak the sampler. Robert J Goedman coded all the models of the excellent “Statistical Rethinking” book in DynamicHMCModels.jl, which is the most comprehensive collection of examples for this package, and also provides and extremely useful test suite for it. DiffEqBayes.jl included DynamicHMC as a backend, while Soss.jl provides a higher-level DSL for building models. Users (who are, mostly, also the authors) of these packages provided a lot of example models and test cases.

Changes in 2.0

Documentation

The documentation was rewritten from scratch, now including a worked example. All functions of the API have extensive docstrings, usually with examples where relevant. This documentation should be the starting point for using this package.

Of course, if you have questions, feature requests, or bug reports, don't hesitate to open an issue; I would like to emphasize that it is still perfectly fine to open issues just to ask questions. You can also address questions to @Tamas_Papp on the Julia discourse forum.

Main interface

Most people would now call the sampler with

results = mcmc_with_warmup(Random.GLOBAL_RNG, ∇P, 1000)

where ∇P is an object that supports the LogDensityProblems.jl API (basically “give me a log density and its gradient at this position”, withs some bells and whistles). results is a NamedTuple with fields like chain (a vector of positions in $\mathbb{R}^n$, which most users probably need to transform into a collection of constrained parameters), information on tree statistics and the adapted parameters. A detailed worked example is available.

In contrast to the previous API, the random number generator needs to be explicitly provided. This is in preparation for multithreading, encouraging the user to be conscious of RNGs as mutable states; the internals now also follow this approach with no default RNG in the sampling part.

Exposed warmup building blocks

The new API allows fine-grained control over the warmup stages. For example, this is how one would skip local optimization, ask for a dense (Symmetric) metric instead of the default Diagonal, and provide an initial position while at the same time allowing the stepsize to be found and adapted using the default heuristic:

q = ... # some initial position
warmup_stages = default_warmup_stages(; local_optimization = nothing,
                                      M = Symmetric)
κ = GaussianKineticEnergy(5, 0.1)
mcmc_with_warmup(rng, ℓ, N;
                 initialization = (q = q, κ = κ),
                 warmup_stages = warmup_stages)

The warmup_stages above is just a Tuple, with no secret sauce, you are provided the tools to make up your own if necessary.

Changed heuristics and warmup defaults

In contrast to Stan, the old API did not look for a local optimum before starting the stepsize adaptation. This is fine in most cases, but occasionally adapting to an otherwise atypical region of the parameter space can cause problems with the algorithm later on. On the other hand, adapting too eagerly to models with singularities in the log posterior can also backfire very easily.

The new default heuristics make a half-hearted effort to go near some local optimum, but don't overdo it, which I think is the right compromise. Also, there is some protection against singularities on the “edges” of $\mathbb{R}^n$ (which can happen with the “funnels” of hierarchical models).

Stepsize adaptation became a bit more robust with some tweaks. I still think that the initial bracketing algorithm in this package is better than Stan's in some cases and worth the extra couple of evaluations, as it plays nicer with the dual averaging for posteriors where the optimal stepsize can change rapidly.

The default kinetic energy metric is now Diagonal. This should be nearly optimal except for very heavily correlated posteriors which also happen to be devoid of any other pathologies (this is rare in practice).

Diagnostics

Diagnostics were reorganized into a DynamicHMC.Diagnostics submodule of their own. They are intended for interactive use, and the exposed API can change with just a minor version increment. You can import them into your current module with

using DynamicHMC.Diagnostics

Note that this package has no plotting functionality. Use your favorite plotting package to visualize information, I made the plots below with PGFPlotsX.jl (and relevant code may eventually end up in a mini-package, for now see the link below).

download code as plots.jl

Detailed tree statistics

Each result that contains a chain also comes with corresponding tree_statistics, which is a vector of statistics for each NUTS step. It contains information about acceptance ratios, number of leapfrog steps and tree depth, and the doubling directions. In 2.0, it also contains a field which informs the user about the location (in steps relative to the starting point).

There is a summarize_tree_statistics function that produces a useful summary about acceptance rations:

julia> results = mcmc_with_warmup(Random.GLOBAL_RNG, ℓ, 1000;
                                  reporter = NoProgressReport());

julia> results.tree_statistics[1]
DynamicHMC.TreeStatisticsNUTS(-1.7246438302263802, 2,
                              turning at positions 1:4, 0.963443025058039,
                              7, DynamicHMC.Directions(0x595b1b9c))

julia> summarize_tree_statistics(results.tree_statistics)
Hamiltonian Monte Carlo sample of length 1000
  acceptance rate mean: 0.94, 5/25/50/75/95%: 0.75 0.92 0.97 1.0 1.0
  termination: divergence => 0%, max_depth => 0%, turning => 100%
  depth: 0 => 0%, 1 => 14%, 2 => 55%, 3 => 21%, 4 => 10%, 5 => 0%

Calculating leapfrog trajectories

A call not unlike

traj = leapfrog_trajectory(ℓ, [0, -2], 0.2, -6:10;
                           κ = GaussianKineticEnergy(2),
                           p = [2, 2])

was used to produce the information that was used for the plot below. The resulting vector (here, traj) contains a NamedTuple of positions, momenta, and relative energy. Here the stepsize $\epsilon = 0.2$ was selected to on purpose as near-but-not-quite-unstable, starting from position [0, -2], taking 6 leapfrog steps backward and 10 forward.

The plot below visualizes the energy relative to the starting point.

Exploring log acceptance ratios

It can be very useful to explore log acceptance ratios. explore_log_acceptance_ratios returns a matrix of them with random momenta. In the plot below, we can see things become iffy for $\epsilon > 0.5$, approximately.

Internal changes

Although there is a new API, the bulk of the changes were internal: resulting in (hopefully) much cleaner and more generic code, better unit tests, and improved documentation documentation for the internals, which are especially relevant for users using this package for research. If this affects you, please read the code and the docstrings and feel free to ask questions.

PSA: breaking API changes LogDensityProblems and DynamicHMC

2019-07-25T10:32:13+02:00

The API of LogDensityProblems and DynamicHMC is being redesigned (issues: DynamicHMC#30, LogDensityProblems#45). This is necessary to address some issues related to ease of use, debugging, robustness, adaptation, and better integration with some advanced AD packages like Zygote.

Once the changes are complete, I will post a detailed writeup for the transition. But in the meantime, if you are using my MCMC libraries, you may want to pin versions, or put the following in your Project.toml's [compat] section (you only need the [compat] if there isn't one):

[compat]
DynamicHMC = "^1.0.5"
LogDensityProblems = "^0.8.3"

For more information, please refer to the Pkg docs.

Changes for DynamicHMC.jl and related packages

2018-10-04T13:30:24+02:00

TL;DR: new wrapper types for AD (with support for Flux.jl), new transformation interface, see the examples.

DynamicHMC.jl is Julia package for Hamiltonian MCMC using the NUTS algorithm. Its interface and the related packages underwent some reorganization which involves API changes. This post is a brief summary, as I prefer to maintain the actual examples in a repository I can test. You are encouraged to read these, and the documentation for each package (some of which is WIP).

A summary of the changes:

TransformVariables.jl replaces the previous DSL for transforming parameters (which was ContinuousTransformations.jl). It has a different syntax, and should make it easier to specify custom transformations in the future (this is work in progress).

For example,
```
as((μ = asℝ, σ = asℝ₊))
```
will specify a transformation of 2 real numbers to a NamedTuple with fields μ (unconstrained) and σ (constrained to be positive, eg a standard deviation).
LogDensityProblems.jl contains the wrapper types for log density functions $\mathbb{R}^n \to \mathbb{R}$. The most common workflow is making your function map a tuple of parameters to a ::Real (which may be -Inf), define a transformation, and then obtain the gradient via automatic differentiation. Currently ForwardDiff.jl (suboptimal for gradients, but very robust) and Flux.jl (faster) are supported. Support for Zygote.jl and ReverseDiff.jl should be easy to add, just open an issue.

For example, once you have specified a callable p that takes the transformed parameters which result from transformation t,
```
P = TransformedLogDensity(t, p)
∇P = FluxGradientLogDensity(P)
```
will wrap them so that Flux.jl is used for obtaining the gradient via AD.
DynamicHMC.jl expects a callable that supports the dimension query function, takes a vector of reals, and returns a LogDensityProblems.ValueGradient type. The recommended way of obtaining this is using the wrapper libraries above, but you are of course free to code your own.

To continue the example, obtaining samples is now as simple as
```
chain, NUTS_tuned = NUTS_init_tune_mcmc(∇P, 1000);
```
because ∇P now contains the dimension information.

All of the above packages are registered. The new version for DynamicHMC.jl is v1.0.0, as the API change is breaking. All of the packages require at least Julia v0.7, but v1.0 or later is recommended.

DynamicHMCExamples.jl has some examples on suggested usage. I am happy to provide support with coding specific models, just please open an issue there. As usual, bug reports, feature requests and PRs are welcome.

Finally, I would like to thank users for their patience and feedback. I hope that you will find new interface more convenient, and the improved speed useful.

PS.: I have removed the Disqus comments from the blog; they broke the style and hardly anyone was using them anyway.

Making dotted graph paper in Julia with Luxor.jl

2018-06-18T11:30:24+02:00

I needed some graph paper with subdivisions at centimeters and millimetres. There are some generators online, but I found their output too heavy, visually cluttered, or simply not ideal (yes, I am very picky about these things).

The obvious solution was to generate a PDF or SVG using a program. Before learning Julia, I would have just started in pgf/tikz, and spent hours searching online to do the thing I want. pgf/tikz are fine tools, but I generally try to avoid programming anything nontrivial in LaTeX, so I always have to spend a lot of time debugging my code.

However, being a Julia user, this was a great opportunity to try Luxor.jl, the “Cairo for tourists!”. This means that I don't have to immerse myself in the intricacies of its syntax and concepts; I can just dip my toes and get the results.

I was not disappointed: within 15 minutes of Pkg.add("Luxor"), I had arrived at a solution I like. I am very satisfied with Luxor.jl and will probably use it in the future when I need graphics built up from primitives.

download as dotted_paper.jl

using Luxor

#######################
# customize script here
#######################
paper = "A4"                       # paper size
major_unit = 1cm                   # gridlines at each
subdivisions = 10                  # dots on gridlines in between
radius = 0.1mm                     # radius for the tiny dots
margins = (5mm, 5mm)               # minimum margins (may be more, centered)
color = "black"                    # color for the tiny dots
filename = "dotted_paper.pdf"      # output goes here, OVERWRITTEN

#######################################################
# runtime code - you probably don't need to change this
#######################################################
pagesize = paper_sizes[paper]
nx, ny = @. floor(Int, (pagesize - (2 * margins)) / major_unit)
offsetx, offsety = @. margins + (pagesize % major_unit) / 2

major_grid_x = offsetx .+ major_unit * (0:nx)
major_grid_y = offsety .+ major_unit * (0:ny)
minor_grid_x = offsetx .+ (major_unit / subdivisions) * (0:(nx*subdivisions))
minor_grid_y = offsety .+ (major_unit / subdivisions) * (0:(ny*subdivisions))

Drawing(paper, filename)
background("white")
sethue(color)
@. circle(Point(major_grid_x, minor_grid_y'), radius, :fill);
@. circle(Point(minor_grid_x, major_grid_y'), radius, :fill);
finish()
preview()

The final PDF is here.

Teaching

2018-04-27T14:48:06+02:00

Introduction to Bayesian estimation of DSGE models

PDF Slides (with references and further reading) for an introductory course on Bayesian estimation of DSGE models. Mostly Bayesian concepts and practical issues (diagnostics, posterior predictive checks, convergence problems).

Common random variables: an optimization case study

2018-03-23T15:23:16+01:00

Code used in this post was written for Julia v0.6.2 and, when applicable, reflects the state of packages used on 2018-03-23. You may need to modify it for different versions.

Motivation

When simulating models with random components, it is frequently advantageous to decompose the structure into

parameters $\theta$ that we need to characterize structural relationships,
noise $\varepsilon$ that is independent of parameters,
a function $f(\theta, \varepsilon)$ that generates observed data or moments.

Doing this allows us to use common random variables (aka common random numbers), which is a technique which simply keeps $\varepsilon$ fixed for different $\theta$. This can help with making $f$ differentiable, which allows the use of derivative-based optimization algorithms (eg for maximum likelihood or MAP) or derivative-based MCMC methods. It is also used to reduce the variance of simulated moments.

When implementing this technique in Julia, I frequently create a wrapper structure that saves the $\varepsilon$, allocating an Array which can be updated in place. Since a redesign of DynamicHMC.jl is coming up which will accommodate simulated likelihood methods in a more disciplined manner, and I wanted to explore other options, most importantly StaticArrays.jl.

Here I benchmark the alternatives for Julia v0.6.2 using a simple toy model. TL;DR for the impatient: StaticArrays.jl is 150x faster, and this does not depend on using immutable or mutable StaticArrays, or even reallocating every time new $\varepsilon$s are generated.

Problem setup

The setup is simple: we draw $M$ observations, and our noise is

\[ \varepsilon_{i,j} \sim N(0, 1) \qquad \text{for } i = 1, \dots, M; j = 1, \dots, 7. \]

Our parameters are $\mu$ and $\sigma$, and for each $i$ we calculate

\[ A_i = \frac17 \sum_{j=1}^7 \exp(\mu + \sigma \varepsilon_{i,j}) \]

which is the sample average after a nonlinear transformation. The $7$ is a bit accidental, it comes from simplifying an actual problem I am working on. We are interested in the sample average for $A_i$. I deliberately refrain micro-optimizing each version, to reflect how I would approach a real-life problem.

We code the common interface as

using BenchmarkTools
using StaticArrays
using Parameters

# common interface

"Dimension of noise ``ϵ`` for each observation."
const EDIM = 7

"""
Common random variables. The user needs to define

1. `observation_moments`, which should use `observation_moment`,
2. `newcrv = update!(crv)`, which returns new common random variables,
potentially (but not necessarily) overwriting `crv`.
"""
abstract type CRVContainer end

observation_moment(ϵ, μ, σ) = mean(@. exp(μ + σ * ϵ))

average_moment(crv::CRVContainer, μ, σ) = mean(observation_moments(crv, μ, σ))

"Calculate statistics, making `N` draws, updating every `L`th time."
function stats(crv, μ, σ, N, L)
    _stat() = (N % L == 0 && (crv = update!(crv)); average_moment(crv, μ, σ))
    [_stat() for _ in 1:N]
end

The way I wrote stats is representative of how I use HMC/NUTS: simulated moments on the same trajectory are calculated with the same $\varepsilon$s, which are updated for each trajectory. Of course, the parameters would change along the trajectory; here they don't, but this does not affect the benchmarks.

The semantics of update! allows both in-place modifications and a functional style.

Using a preallocated matrix

This is the standard way I would write this.¹ $\varepsilon$s are in columns of a matrix, which is preferable for mapping them as slices, then they are mapped using observation_moment.

update! overwrites the contents.

"""
Common random variables are stored in columns of a matrix.
"""
struct PreallocatedMatrix{T} <: CRVContainer
    ϵ::Matrix{T}
end

PreallocatedMatrix(M::Int) = PreallocatedMatrix(randn(EDIM, M))

update!(p::PreallocatedMatrix) = (randn!(p.ϵ); p)

observation_moments(p::PreallocatedMatrix, μ, σ) =
    vec(mapslices(ϵ -> observation_moment(ϵ, μ, σ), p.ϵ, 1))

Using static vectors

We share the following between various static vector implementations:

"Common random variables as a vector of vectors, in the `ϵs`."
abstract type CRVVectors <: CRVContainer end

observation_moments(p::CRVVectors, μ, σ) =
    map(ϵ -> observation_moment(ϵ, μ, σ), p.ϵs)

I find the use of map more intuitive here than mapslices above.

Static vectors, container preallocated

In the first version using static vectors, we keep SVector in a Vector, and update in place.

struct PreallocatedStaticCRV{K, T} <: CRVVectors
    ϵs::Vector{SVector{K, T}}
end

PreallocatedStaticCRV(M::Int) =
    PreallocatedStaticCRV([@SVector(randn(EDIM)) for _ in 1:M])

function update!(p::PreallocatedStaticCRV)
    @unpack ϵs = p
    @inbounds for i in indices(ϵs, 1)
        ϵs[i] = @SVector(randn(EDIM))
    end
    p
end

Mutable static vectors, overwritten

We modify this to use mutable vectors — this should not make a difference.

struct MutableStaticCRV{K, T} <: CRVVectors
    ϵs::Vector{MVector{K, T}}
end

MutableStaticCRV(M::Int) =
    MutableStaticCRV([@MVector(randn(EDIM)) for _ in 1:M])

function update!(p::MutableStaticCRV)
    @unpack ϵs = p
    @inbounds for i in indices(ϵs, 1)
        randn!(ϵs[i])
    end
    p
end

Static vectors, allocated each time

Finally, for the third solution,

struct GeneratedStaticCRV{K, T} <: CRVVectors
    ϵs::Vector{SVector{K, T}}
end

GeneratedStaticCRV(M::Int) =
    GeneratedStaticCRV([@SVector(randn(EDIM)) for _ in 1:M])

update!(p::GeneratedStaticCRV{K, T}) where {K, T} =
    GeneratedStaticCRV([@SVector(randn(T, K)) for _ in indices(p.ϵs, 1)])

Results

Running

@btime mean(stats(PreallocatedMatrix(100), 1.0, 0.1, 100, 10))
@btime mean(stats(PreallocatedStaticCRV(100), 1.0, 0.1, 100, 10))
@btime mean(stats(MutableStaticCRV(100), 1.0, 0.1, 100, 10))
@btime mean(stats(GeneratedStaticCRV(100), 1.0, 0.1, 100, 10))

we obtain

solution	time	allocations
`PreallocatedMatrix`	`230 ms`	`22 MiB`
`PreallocatedStaticCRV`	`1.5 ms`	`102 KiB`
`MutableStaticCRV`	`1.5 ms`	`104 KiB`
`GeneratedStaticCRV`	`1.5 ms`	`666 KiB`

As a preview of future improvements, I tried PreallocatedMatrix on current master (which will become Julia v0.7, obtaining 3.5 ms (2.46 MiB), which is really promising.²

The conclusion is that StaticArrays simplifies and speeds up my code at the same time. I especially like the last version (GeneratedStaticCRV), because it obviates the need to think about types in advance. While here the example is simple, in practice I would use automatic differentiation, which makes it more challenging to determine buffer types in advance. I expect I will transition to a more “buffer-free” style in the future, and design the interface for DynamicHMC.jl accordingly.

download code as code.jl

PS: From now on, my blog posts with Julia code will have a banner about version information.

Which will change following this blog post 😁 ^[return]
The other 3 options are slow because of deprecation warnings. ^[return]

Local test coverage in Julia

2018-03-09T08:31:23+01:00

While codecov.io and coveralls.io are fine services and very easy to set up for Github repositories with Julia packages, they become more difficult to use under some circumstances. For example, if testing requires large binaries you don't want to build every time, you need to build a Docker image to run testing and obtain coverage information. Also, if your test scripts take a long time and you have access to a powerful computer, you might want to just run coverage locally when working on the code.

Naturally, Julia has all the facilities for generating and collecting coverage information locally. To make their application even easier, I packaged up a small set of scripts in LocalCoverage.jl that I use to generate and visualize coverage information for Julia packages on my machine.

This is how it works:

run generate_coverage(pkg) to

a. run Pkg.test(pkg; coverage = true),

b. use Coverage.jl for collecting information to a single file in lcov format,

c. unless disabled, run the external program genhtml to format everything as a nice HTML file.
if you have generated the HTML files, open_coverage(pkg) opens them for you in your default browser.
if you want to tidy up, run clean_coverage(pkg).

This is what it looks like for this package:

Setting up Julia with continuous integration on Gitlab

2018-03-08T17:06:54+01:00

As an academic, I picked up good practices for programming mostly by osmosis. My approach to “testing” software went through the following stages:

See if the code runs (this got me through my undergrad years).
Check if the results “look OK”, then forget about testing.
Compare results occasionally to known results from papers or other code (eg in a different language).
Write some unit tests ex post, as an afterthought after the code is finished (time pressure helps to ensure that overtesting is never a problem).
Use unit tests from the very beginning, especially before optimizing and refactoring code.
Set up automatic testing, as part of continuous integration.

I think that 1–3 above is a pretty usual path, naturally traversed after the recognition that some testing would be nice, but lacking the knowledge of how to implement it in a consistent manner. This is comparable to using copies of directories as crude attempts at “version control”.

Later, I picked up 4–6 while being exposed to these ideas when learning about new languages. Automated unit testing is one of those things one does not miss until learning about it, then subsequently cannot imagine doing without. In a research context, the two main advantages are scientific integrity — I should make a best effort to ensure that my results are correct — and dealing with bugs early. While the first one is abstract and philosophical, the second is a practical concern: I found that if I skimp on testing, the bugs show up much later, usually at an inconvenient time, and I will have to spent time locating the bug (not always trivial, especially with numerical code) and switch context to something I was working on months ago. It is my experience that while tests can be tedious to write, time spent on them is a very good investment.

I knew about unit tests before coming to Julia, but learned about automated CI in the Julia community. This was because package template libraries “do the right thing” by making it very easy to set up an automated testing framework: for example, PkgDev.generate creates the appropriate test configuration for Travis CI and various coverage services.

I never cease to be amazed by the fact that these services are available for free for public / open source projects, which is very generous of these providers. However, occasionally one would like to keep the project private for a little while. The usual scenario for me is working on code that is related to a paper, which I plan to make public with the latter; in this case one would need the pro (non-free) version of Travis and related tools.

Alternatively, Gitlab offers CI/CD with private repositories. I am exploring this at the moment for various projects, and boiled down the necessary configuration into the repository GitlabJuliaDemo.jl. It has

a CI setup for Pkg.test,
a coverage summary as a percentage.

While coverage analysis could be automated too with a custom Docker image, I leave his for future work.¹

So far, I am very satisfied with Gitlab. The interface is well-designed, clean, and intuitive; tests complete in a few minutes (just like Travis).

In the next post I will talk about local coverage analysis in Julia. ^[return]

Continuous integration for Julia packages using Docker

2018-01-19T08:47:06+01:00

This post may be useful for maintainers of Julia packages which require a large binary dependencies on CI services like Travis.

I have recently started using Kristoffer Carlsson's excellent PGFPlotsX for plotting. The package is a thin wrapper which emits LaTeX code for use with pgfplots, which is extremely versatile and well-documented.¹ However, since most of the action happens in LaTeX, unit testing requires a lot of binary dependencies, including the TeXLive suite and some related packages. This is not a problem on one's own machine where these would need to be installed just once, but when I submitted PRs, tests on Travis timed out more often than not because it had to install all of these for every run using apt-get.

The documentation of Travis suggested that docker may be a solution for such cases, and I have been looking an opportunity to experiment with it anyway. After reading their tutorial it was relatively quick to produce an image based on plain vanilla Ubuntu 17.10, which is available as a docker image to build on, and the required TeXLive and related packages, plus some utilities.

During building the image, I download the binaries for the stable version Julia, while nightly is downloaded on demand. This speeds up CI by 40–50 seconds for stable.

This is how it is run:

the directory of the Julia package is mounted in the container at /mnt,
Pkg.clone() and testing proceed as usual,
coverage results are copied back to /mnt when done.

The resulting image runs in 3–4 minutes consistently. In case someone finds it useful for Julia packages with similarly large binary dependencies, I made it available as texlive-julia-minimal-docker on Github.² Naturally, for projects with other large binary dependencies, one would install different Ubuntu packages or binaries.

Using this package accelerated my plotting workflow in Julia. A post on this will follow soon. ^[return]
“Minimal” turns out to be a misnomer, since some dependencies end up requiring X11 and the image is >700GB. ^[return]

Working with large Julia source files in Emacs

2017-12-04T15:32:41+01:00

When writing software, especially libraries, a natural question is how to organize source code into files. Some languages, eg Matlab, encourage a very fragmented style (one function per file), while for some other languages (C/C++), a separation between the interface (.h) and the implementation (.c/.cpp) is traditional.

Julia has no such constraint: include allows the source code for a module to be organized into small pieces, possibly scattered in multiple directories, or it can be a single monolithic piece of code. The choice on this spectrum is up to the authors, and is largely a matter of personal preference.

When I started working with Julia, I was following the example of some prominent packages, and organized code into small pieces (~ 500 LOC). Lately, whenever I refactored my code, I ended up putting it in a single file.

I found the following Emacs tools very helpful for navigation.

Form feeds and page-break-lines-mode

Form feed, or \f, is an ASCII control character that was used to request a new page in line printers. Your editor may display it as ^L. It has a long history of being used as a separator, and Emacs supports it in various ways.

By default, C-x [ and C-x ] take you to the previous and next form feed separators. Combined with numerical prefixes, eg C-3 C-x [, you can jump across multiple ones very quickly. Other commands with page in their name allow narrowing, marking, and other functions.

Many Emacs packages provide extra functionality for page breaks. My favorite is page-break-lines, which replaces ^L with a horizontal line, so that the output looks like this:

export
    ML_estimator

# general API

"""
    ML_estimator(ModelType, data...)

Estimate `ModelType` using maximum likelihood
on `data`, which  is model-specific.
"""
function ML_estimator end

Finding things quickly

I am using helm pervasively. helm-occur is very handy for listing all occurrences of something, and navigating them. The following is an except from base/operators.jl, looking for isless:

operators.jl:213:types with a canonical total order should implement `isless`.
operators.jl:227:<(x, y) = isless(x, y)
operators.jl:300:# this definition allows Number types to implement < instead of isless,
operators.jl:302:isless(x::Real, y::Real) = x<y
operators.jl:303:lexcmp(x::Real, y::Real) = isless(x,y) ? -1 : ifelse(isless(y,x), 1, 0)

You can move across these matches, jump to one in an adjacent buffer while keeping this list open, or save the list for later use. Its big brother helm-do-grep-ag is even more powerful, using ag to find something in a directory tree.

With these two tools, I find navigating files around 5K LOC very convenient — the better I learn Emacs, the larger my threshold for a “large” file becomes.¹

In case you are wondering, the largest files are around 6K LOC in Julia Base at the moment. ^[return]

Publication quality plots in Julia

2017-12-01T09:56:00+01:00

In light of recent discussions on Julia's Discourse forum about getting “publication-quality” or simply “nice” plots in Julia, I thought it would be worthwhile to briefly summarize what works for me.¹ If you are a seasoned Julia user, this post may have nothing new for you, but I hope that newcomers to Julia find it useful.

Generate the data

I try to separate data generation and plotting. The first may be time-consuming (some calculations can take hours or days to run), and I find it best to save the results independently of any plotting. Recently I was sitting at a conference where a presentation about a really interesting topic had some plots that were extremely hard to see: if I remember correctly, something like 10x2 subplots, with almost all fine detail lost due to the resolution of the projector or the human eye. When someone in the audience asked about this, the presenting author replied that he is aware of the issue, but remaking the plots would involve rerunning the calculations, which take weeks. Saving the data separately will ensure that you are never in this situation; also, you can benefit from updates to plotting libraries when tweaking your plots.

For saving results, JLD2 is probably the most convenient tool: while it is technically work in progress, it is stable, fast, and convenient.² The key question is where to save the data: I find it best to use a consistent path that you can just include in scripts.

You have several options:

define a global variable in your ~/.juliarc for your projects, and construct a path with joinpath,
if you have packaged your code, Pkg.dir can be used to obtain a subdirectory in the package root,
if your code is in a module, you can wrap @__DIR__ in a function to obtain a directory.

For this blog post I used the first option, while in practice I use the second and the third.

To illustrate plots, I use the code below to generate random variates for sample skewness, and save it.

download as data.jl

using StatsBase                 # for skewness
using JLD2                      # saving data
cd(joinpath(BLOG_POSTS, "plot-workflow")) # default path
sample_skewness = [skewness(randn(100)) for _ in 1:1000]
@save "data.jld2" sample_skewness # save data

Make the plot

No plotting so far, so let's remedy that. I use Plots.jl, which is a metapackage that unifies syntax for plotting via various plotting backends in Julia. I find this practical, because I can quickly switch backends for different purposes, and experiment with various options when I find the output suboptimal. The price you pay for this flexibility is compilation time, a known issue which means that you have to wait a bit to get your first plot. Separating plotting and data generation has the advantage that once I fire up the plotting infrastructure, I switch to “plotting mode” and clean up several plots at the same time.

Users frequently ask what the “best” backend is. This all depends on your needs, but these days I use the pgfplots() backend almost exclusively.³ The gr() backend is also useful, because it is very fast.

Time to tweak the plot! I find the attributes documentation the most useful for this. For this plot I need axis labels, a title, and prefer to disable the legend since I am plotting a single series. I am also using LaTeXStrings.jl, which means that I can use LaTeX-compatible syntax for labels seamlessly (notice the L before the string).

download as plot.jl

using JLD2                      # loading data
using Plots; pgfplots()         # PGFPlots backend
using LaTeXStrings              # nice LaTeX strings
cd(joinpath(BLOG_POSTS, "plot-workflow")) # default path
@load "data.jld2"                         # load data
# make plot and tweak; this is the end result
plot(histogram(sample_skewness, normalize = true),
     xlab = L"\gamma_1", fillcolor = :lightgray,
     yaxis = ("frequency", (0, 2)), title = "sample skewness", legend = false)
# finally save
savefig("sample_skewness.svg")  # for quick viewing and web content
savefig("sample_skewness.tex")  # for inclusion into papers
savefig("sample_skewness.pdf")  # for quick viewing

Having generated the plot, I save it in various formats with savefig. The SVG output is shown below.

How to get help

If you cannot achieve the desired output, you can

reread the Plots.jl manual,
study the example plots,
ask for help in the Visualization topic.

For the third option, make sure you include a self-contained minimal working example,⁴ which also generates or loads the data, so that others can run your code as is. Randomly generated data should be fine, or standard datasets from RDatasets.jl.

Sometimes you will find that the feature you are looking for is not (yet) supported. You should check if there is an open issue for your problem (the discussion forum linked above is useful for this), and if not, open one.

When asking for help or just discussing plotting libraries in Julia, please keep in mind that they are a community effort with volunteers devoting their time to address a very difficult problem. Plotting is not a well-defined exercise, it involves a lot of heuristics and special cases, and most languages took years to get it right (for a given value of “right”). Make it easy for people to help you by making a reproducible, clean MWE: it is very hard to explain how to improve your plot without the actual code and output.

Code in this post was written in December 2017, you may need to tweak it if the API of the packages changes. ^[return]
In the worst case scenario, you can always regenerate the data ☺ ^[return]
Note that you need a working TeX installation, which is easy to obtain on Linux. ^[return]
You should use triple-backticks ``` to format your code. ^[return]

Checking uncommitted changes when opening a file in Emacs

2017-11-18T10:48:30+01:00

Alas, the following happens all too frequently: I am working on code, some interruption happens, and I fail to commit the changes coherently into the repository.

Next time I open the file, perhaps to work on some other feature, I forget that I have uncommitted changes and work on something new. When staging, I realize the mistake and have to spend time disentangling the mess.

The following Emacs Lisp snippet takes care of this problem by checking for uncommitted changes and taking me to the magit popup if there is something I should deal with:

(cl-defun tkpapp/check-file-and-popup (&optional (file (buffer-file-name)))
  "If the file is version controlled with git and has uncommitted
changes, open the magit status popup."
  (require 'magit-core)
  (when (and file (magit-anything-modified-p t file))
    (magit-status)))

(add-hook 'find-file-hook 'tkpapp/check-file-and-popup)

WIP: making error locations in julia-repl clickable

2017-11-01T12:27:23+01:00

I scratched a long-standing itch and made locations in error messages "clickable" in julia-repl. Not yet merged into master, the change is in the clickable-locations branch.

Testing is needed because of some hacks (again, I am not an Emacs expert), I will see if there are issues then merge it. This is what it looks like, those red and orange lines take you to the source:

julia> include("/tmp/Foo.jl")
ERROR: LoadError: UndefVarError: T not defined
Stacktrace:
 [1] include_from_node1(::String) at ./loading.jl:576
 [2] include(::String) at ./sysimg.jl:14
while loading /tmp/Foo.jl, in expression starting on line 9

julia>

Working with a large ragged dataset in Julia

2017-10-26T19:14:07+02:00

Introduction

One of the projects I am working on at the moment at the Institute for Advanced Studies (in Vienna) is an analysis of the Austrian Social Security Database. We are using this extremely rich dataset to refine our understanding of the cross-sectional heterogeneity and age structure of employment and labor market participation.¹

Naturally, I am using Julia, not only because it is fast, convenient, and elegant, but also because it allows me to use a single language for data processing, exploratory data analysis, descriptive statistics, and more sophisticated Bayesian indirect inference using MCMC. When analyzing real-world data, it is useful to do some exploratory plots, fit a simple model, refine, disaggregate, fit a more complex model, and repeat this until I am satisfied with the result. Not having to switch languages is a great bonus.

In this post I talk about my experience with the very first step of the above process: ingesting and collating a large, ragged dataset using Julia. I found that accomplishing this was nontrivial: while the DataFrames.jl ecosystem is converging nicely, I found that I had to develop some custom tools to work with this dataset. I hope that others in a similar situation find them and this writeup useful.

I made the resulting libraries available on Github, with some documentation and lots of unit tests, but they are not finalized since I will probably rewrite some of them once named tuples are incorporated into the language. This is not a detailed introduction to any of these libraries, especially since they are subject to change, rather an account of work in progress and some starting points if you want to do something similar. However, if you want to use these libraries, look at the docstrings and the unit tests, and feel free to ask for help, preferably by opening an issue for the relevant library.

About the data

The whole data comprises about 2000 million observations on various "spells", which either involve contributions to or benefits from Austria's comprehensive social security system (eg being employed, or being on maternity leave). Each spell has the unique ID of the individual it belongs to, a start and end date, and spell information (eg insurance event). They look something like this:²

1;...;19800101;19800201;...;AA;...;
1;...;19800301;19800401;...;B;...;
2;...;19800701;19800901;...;CCC;...;

Fields are separated by ;, the ...s are fields which we ignore for now (we may use them later). Dates have the yyyymmdd format. There are about 70 spell types.

Gzipped data in delimited format is about 45 GB, raw data would be over 500 GB. Data for each year is in a separate file, so individual 1 may have spells scattered in multiple files. Ideally, for our analysis, we would like to end up with a data structure that has the spells organized by individual, eg

individual 1
- start date 1980-01-01, end date 1980-02-01, spell type AA
- start date 1980-03-01, end date 1980-04-01, spell type B
- ... other spells from subsequent files
individual 2
- ...

The number of spells varies by individual, which makes this dataset ragged.³

Rationale for custom tools

A simple back of the envelope calculation is useful to think about the size of the dataset. If we encode each column as an Int64 or similar (eg Date), we use

\[8 \text{ bytes} \times 2 \cdot 10^9 \approx 16 \text{ gigabytes}\]

per column. For 4 columns, this would use up 64 GB, plus some extra for keeping track of the ragged structure. While this is feasible if we have enough RAM, it is very nice to use smaller machines (especially laptops — I am typing this on the train) for exploratory data work.⁴ Also, economizing on RAM speeds up the calculations, as more data fits into the CPU cache.

Early experiments suggested that simply reading this data into native Julia structures or tools in the DataFrames.jl ecosystem is either infeasible or unnecessarily slow. I also considered databases, but found them to be more trouble than it is worth, especially since what I am doing below is straightforward and fits into Julia very nicely.

Below, I discuss the key ingredients I used to process this dataset.

Mmapping large columns

Julia supports memory mapped arrays, which map virtual memory to disk seamlessly, allowing lazy loading and access managed by the virtual memory manager. Using the syntax

io = open("path_to_file.bin", "w+") # create, truncate
A = Mmap.mmap(io, Vector{Int}, 200) # map to array

gives you a vector that is mapped to the disk (you have to call Mmap.sync! after you are finished to make sure, and you have to use growth = true, which is the default). The advantage is that the size of the array is limited by the disk space, not RAM, and the OS takes care of reading, writing, and caching as necessary.

A complication for our data analysis is that we do not know the total number of elements before having read the whole dataset, so we can't specify the dimensions above. Fortunately, simply opening a stream and writeing values of bits types works fine.⁵

I packaged the code for managing columns of bits types using the strategies above in LargeColumns.jl, which keeps track of column types and imposing some basic sanity checks. This makes working with large vectors as easy as

using LargeColumns
cols = MmappedColumns("path/to/directory", 2_000_000_000, Tuple{Float64, Int})

which takes care of mmaping, and makes cols behave as a vector of tuples of the given type. The files, including metadata, are located in the given directory.

Ragged data and collation

I want to end up with data grouped by individuals contiguously, eg

1 1 1 1 2 2 2 3 3 4 4 ...

where the first 4 observations belong to individual 1, the second 3 to 2, and so on. This can be indexed with UnitRanges: for individual 1 we would use 1:4, for 2, 5:8, etc. Note that storage of both endpoints is unnecessary, since they can be calculated from a cumulative sum, also, we can reuse the same index for multiple columns.

The package RaggedData.jl implements simple datastructures for counting, collating, and indexing ragged data into vectors. When I first parse and ingest the data, I count the number of observations for each individual:

id_counter = RaggedCounter(Int32, Int32)

while !eof(...) # process by line
    id = parse_id_from_line(...)
    push!(id_counter, id)
end

Then in the second pass, I start with the index for the first observation for each (eg 1, 5, 9 above), and increment it for each row of the data. So the first observation for individual 2 goes to index 5, the second 6, and so on. For this, I create a collator object coll:

coll, ix, id = collate_index_keys(id_counter, true);

for i in indices(first_pass, 1)
    id, spell_index, dates = first_pass[i]
    j = next_index!(coll, id)
    collated[j] = (spell_index, dates)
end

where collated is another set of mmaped columns. ix is used later for indexing into the result: ix[1] gives the UnitRange for the observations about the first individual, and so on.

Saving space

Before processing the whole dataset, I assumed that the individual ids fit into Int32s. This is easy to verify after parsing, with the standard constructor, which simply throws an error if the value does not fit:

julia> Int32(typemax(Int64))
ERROR: InexactError()
Stacktrace:
 [1] Int32(::Int64) at ./sysimg.jl:77

For the spell types, I simply indexed them in the order of appearance, saving the index as an Int8. They are reconstructed using IndirectArrays.jl when working with the data.

Dates turned out to be the trickiest, until I realized that using Int16, I can represent a timespan of about 179 years, which is a lot more than I need for this data. Of course, this requires that we count from some epoch other than 0001-01-01 like Base.Date. Fortunately, Julia allows encoding the epoch in the type, making it costless as long as I use it consistently for the same dataset. The package FlexDates.jl implements this approach.

Parsing

Being very impressed by the amazing speed gains for date parsing in Julia (see #18000, #15888, #19545), I used this project as an excuse to experiment with parsers. Existing packages like TextParse.jl are already so fast that writing yet another parser library would not have made sense for a small amount of data, but since I plan to reuse this code for large datasets I felt the investment was justified.

Most of the datasets I work with are ASCII: other character sets are still very rare in social science data, since data is predominantly anonymous (so no names), categorical variables are usually encoded as short strings or integers, and the rest are numbers and punctuation. Moreover, delimited UTF-8 can be parsed as ASCII when the delimiters themselves are ASCII.

For this dataset, I was also free to ignore quotes and within-field linebreaks, since they do not occur in the data dumps. Given these, I was free to parse this dataset as ASCII, ie UInt8 (bytes). The algorithms are very simple: parse a given number of characters (eg as numbers), or stop when hitting a delimiter.

Since I ignore some fields, I also needed functionality to simply skip to the next delimiter, returning nothing (but the position for the next byte after the delimiter) — this takes about 25–30% of the time, compared to parsing it. The result is packaged as ByteParsers.jl. Parsing using ASCII turns out to be 30%–120% faster than UTF-8.

Putting it all together

I use three passes to process the data.

First pass

The first pass parses the data and writes it out in a binary format, also counting observations for each individual at the same time. Categorical data is indexed in the order of appearance and written out as an Int8, dates are written using FlexDate{Date(2000, 1, 1), Int16}, represented with 16 bits.

Open the gzipped files using CodecZlib.jl. Open sinks for binary data using LargeColumns.jl.
Read and parsed them line by line, using ByteParsers.jl. You can also use TextParse.jl.
Write the parsed data into the sinks, at the same time counting with a RaggedCounter from RaggedData.jl.
Close the streams, write the categorical values and the RaggedCounter using JLD2.jl.

The whole process takes about 90 minutes, and generates 18 GB of binary data.

Second pass

The second pass reads back the binary dump from the first pass using mmap, and collates observations for the individuals it using a RaggedCollate indexer from RaggedData.jl. The latter is an object which keeps track of where observations should end up, if their counts are consistent with the first pass. The result is written using LargeColumns.jl into mmapped columns, and it is reasonably fast, taking about 30–80 minutes, depending on the RAM size (the non-contiguous collating process has to use the disk if the resulting large vectors cannot fit in RAM). Finally, the RaggedIndex object is written out using JLD2.

Third pass

The third pass is optional, it sorts spells by the start date for each individual (we found this helps the kind of analysis we perform). It uses the mmaped columns from the second pass, and takes about 2 minutes (since the data is accessed almost linearly).

Using the data

The columns are mmapped using LargeColumns.jl. IndirectArrays.jl is used to reconstitute categorical data with the keys previously saved, and the resulting vector is wrapped in a ragged access data structure using RaggedColumns (from RaggedData.jl) and the previously saved index. Iterating through the dataset takes about 2 minutes.

Conclusion

Ingesting and working with large amounts of data turns out to be very simple and convenient using mmaped arrays in Julia. I packaged the code into libraries because

I like to have unit test, especially if I keep benchmarking and optimizing,
It simplifies code and communication for colleagues I am cooperating with,
May be useful in future projects,
I find packaged code with continuous integration tests closer to the idea of reproducible research.

I plan to register some of these libraries in the future (when the interface stabilizes).

This research is supported by the Austrian National Bank Jubiläumsfonds grant #17378. ^[return]
The samples shown here are made up, the actual dataset is not public. ^[return]
Irregular and non-rectangular are also used. ^[return]
We also used a subset of the data for initial work. ^[return]
Currently needs a workaround for which I submitted a PR. ^[return]

My Julia workflow

2017-10-21T12:40:04+02:00

(edit 2017-10-22: fixed path for PkgDev.generate example)

This is a summary of the workflow I find ideal for working with Julia. Although the manual has a section on workflow, it does not mention all the tools that I find useful, so perhaps this will benefit some users of Julia.

I use Emacs, with julia-mode (for editing the source) and julia-repl (REPL integration). The latter is my own package; you can use ESS instead, which has some advantages (eg multiple inferior processes) and disadvantages (no ANSI terminal support). The choice of an editor is highly subjective: at the end of the day, all you need is one that is capable of sending code to the REPL and can, in turn, be used by the REPL to open a file at a particular point. I use

ENV["EDITOR"] = "emacsclient"

in ~/.juliarc.jl to ensure this. This helps me find code with the @edit macro.

Small code snippets and experiments below ~30 lines just go into files, from which I send regions of code to the REPL. Frequently, for throwaway code, I just open a file in /tmp/, which will get removed automatically after the next reboot.

Even very small projects get their own package. This way I get version control¹ and a sensible structure for unit tests set up automatically. I put my own packages in their own directory, keeping them separate from Pkg.dir(). This allows me to use the same package across Julia versions, and makes Pkg.update() ignore them. I tell Julia where they are with

const LOCAL_PACKAGES = expanduser("~/src/julia-local-packages/")
push!(LOAD_PATH, LOCAL_PACKAGES)

I create local packages with

import PkgDev
PkgDev.generate("MyPkg", "MIT"; path = LOCAL_PACKAGES)

Then I open the file and start working on it with

using MyPkg

I use Revise.jl to automate reloading.² This package has changed my workflow completely; it can cope with most changes, except for type redefinitions. For these, I need to restart the REPL.

To test my code, I use Pkg.test with RoguePkg.jl, which makes it find packages outside Pkg.dir() for testing and benchmarks:

Pkg.test(pkg_for"MyPkg")

I use the amazing magit for interacting with git — having obtained funding on KickStarter recently, it is bound to become even more convenient. ^[return]
You just need to set it up once according to its documentation, after that it is automatic. ^[return]

HTML "screenshots" from Emacs

2017-10-17T13:26:04+02:00

htmlize, written by Hrvoje Nikšić, is a neat little Emacs package that converts face information from an Emacs buffer (or region) into HTML, effectively allowing the verbatim reproduction of what it looks like.

I found this so useful for blogging that I submitted a PR which saves marked up regions as self-contained <pre> snippets, complete with inline CSS. Hrvoje kindly merged the PR today, so now you can call htmlize-region-save-screenshot and the result will be saved into the kill ring. You can paste this into, say, a blog post written in Markdown or Mmark, such as this one, and get a "screenshot" like

(defun htmlize-region-save-screenshot (beg end)
  "Save the htmlized (see `htmlize-region-for-paste') region in
the kill ring. Uses `inline-css', with style information in
`<pre>' tags, so that the rendering of the marked up text
approximates the buffer as closely as possible."
  (interactive "r")
  (let ((htmlize-pre-style t))
    (kill-new (htmlize-region-for-paste beg end)))
  (deactivate-mark))

which is almost exactly what it looks like on my screen when it comes to colors and font style/weight, but using a different font of course. This allows integration of Emacs "screenshots" into blog posts without resorting to pixel-based formats, which would result from taking actual screenshots.

You can install htmlize from MELPA.

Switching from Common Lisp to Julia

2017-10-15T14:17:47+02:00

I have written this post for developers in the Common Lisp community who asked why I am switching to Julia. It may only be relevant for the small set of people who use Common Lisp for scientific computing.

I used Common Lisp for scientific computing for a while, from 2008 to about 2015, in combination with R and C++. This choice may surprise people who don't know about projects like Maxima or FEMLISP, but Common Lisp is not a bad language for scientific computing: it has a great FFI, compilers like SBCL can generate very fast code with a few hints, and the language itself is composed of convenient features that interact nicely.

However, around 2012 I started to become very frustrated with Common Lisp. Despite various attempts, it became very clear that libraries for scientific computing were not goint to take off: there were many one-person efforts (including mine), but very few of them evolved into general tools.

Initially, I was puzzled by this: Common Lisp is an extremely convenient and productive language. Experienced Lisp hackers can write very complex, fast, and elegant libraries in reasonably short time. Why did this not happen for numerical code?

The problem with Common Lisp

Now I think that one of the main reasons for this is that while you can write scientific code in CL that will be (1) fast, (2) portable, and (3) convenient, you cannot do all of these at the same time. Arrays provide a convenient example for this.

Consider

(make-array 5 :element-type 'double-float)

The standard does not guarantee that this gives you an array of double-float: it may (if the implementation provides them), otherwise you get an array of element type T. This turned out to be a major difficulty for implementing portable scientific code in Common Lisp.

However, this gets worse: while you can tell a function that operates on arrays that these arrays have element type double-float, you cannot dispatch on this, as Common Lisp does not have parametric types. For example, if you want to write a sum as

(defmethod mysum ((vec vector))
  (let ((s 0))
    (loop for elt across vec
       do (incf s elt))
    s))

you can dispatch on the argument being a vector, but not on the element type. The compiled code may be generic.

You can of course branch on the array element types and maybe even paper over the whole mess with sufficient macrology (which is what LLA ended up doing), but this approach is not very extensible, as eventually you end up hardcoding a few special types for which your functions will be "fast", otherwise they have to fall back to a generic, boxed type. With multiple arguments, the number of combinations explodes very quickly.

How Julia solves this problem

A comparable native implementation in Julia would be¹

function mysum(vec::AbstractVector{T}) where T
    s = zero(T)
    for elt in vec
        s += elt
    end
    s
end

This is still generic: it works for all subtypes of AbstractVector (including vectors and vector-like objects), but notice how the generated code is conditional on the element type:

julia> @code_warntype mysum([1, 2, 3])
Variables:
  #self#::#mysum
  vec::Array{Int64,1}
  elt::Int64
  #temp#::Int64
  s::Int64

Body:
  begin 
      s::Int64 = 0 # line 3:
      #temp#::Int64 = 1
      4: 
      unless (Base.not_int)((#temp#::Int64 === (Base.add_int)((Base.arraylen)(vec::Array{Int64,1
})::Int64, 1)::Int64)::Bool)::Bool goto 14                                                     
      SSAValue(2) = (Base.arrayref)(vec::Array{Int64,1}, #temp#::Int64)::Int64
      SSAValue(3) = (Base.add_int)(#temp#::Int64, 1)::Int64
      elt::Int64 = SSAValue(2)
      #temp#::Int64 = SSAValue(3) # line 4:
      s::Int64 = (Base.add_int)(s::Int64, elt::Int64)::Int64
      12: 
      goto 4
      14:  # line 6:
      return s::Int64
  end::Int64

julia> @code_warntype mysum([1.0, 2.0, 3.0])
Variables:
  #self#::#mysum
  vec::Array{Float64,1}
  elt::Float64
  #temp#::Int64
  s::Float64

Body:
  begin 
      s::Float64 = (Base.sitofp)(Float64, 0)::Float64 # line 3:
      #temp#::Int64 = 1
      4: 
      unless (Base.not_int)((#temp#::Int64 === (Base.add_int)((Base.arraylen)(vec::Array{Float64
,1})::Int64, 1)::Int64)::Bool)::Bool goto 14                                                   
      SSAValue(2) = (Base.arrayref)(vec::Array{Float64,1}, #temp#::Int64)::Float64
      SSAValue(3) = (Base.add_int)(#temp#::Int64, 1)::Int64
      elt::Float64 = SSAValue(2)
      #temp#::Int64 = SSAValue(3) # line 4:
      s::Float64 = (Base.add_float)(s::Float64, elt::Float64)::Float64
      12: 
      goto 4
      14:  # line 6:
      return s::Float64
  end::Float64

I mentioned "vector-like objects" above, since I can choose different representations for special objects. For example, to do calculations with a vector of 1s, I can define

struct Ones{T <: Number} <: AbstractVector{T}
    len::Int
end

At this point, in order to calculate the sum above, I have two choices:

Implement the relevant interface, with functions like

Base.length(x::Ones) = x.len

and similarly for element access, etc. This would generate specialized code for the method above, reasonably efficient code, but still iterate over the "elements".

In addition, I can define

mysum(vec::Ones{T}) where T = vec.len * one(T)

which would provide a method for mysum.

A sufficiently rich parametric type system with multiple dispatch integrated into the language and supported by a JIT compiler is the secret weapon of Julia. Most of the time, you don't have to do anything, as it happens automatically for concrete types. Sometimes, you have to help the compiler a bit, by writing code where the result is type stable, ie the result type just depends on the type (not the value) of the arguments and can be inferred by the compiler. Sometimes you have to nudge the compiler a bit, and sometimes you have to be careful not to mess up type inference: for example, the zero(T) above gives a 0 of type T, always ensuring a correct type that does not change during the summation.

Comparison of other language features

While I would say that multiple dispatch with parametric types designed into the language from the ground up is the most important feature of Julia, there are other language features worth comparing to Common Lisp.

Metaprogramming is supported. Because of infix syntax, the AST is not as simple as S-expressions, but the tools to work with it are evolving fast. That said, I don't write as many macros as I did in Common Lisp. Parametric types are so powerful that I rarely need macros for performance reasons, and instead of syntax extensions, I often go for zero-cost abstraction via functions and wrapper types. An interesting metaprogramming tool in Julia is generated functions, which allow code generation based on argument templates, I use this frequently. The equivalent of reader macros are called non-standard string literals in Julia.

The foreign function interface of Julia is seamlessly integrated into the language and very convenient to use. Docstrings are almost the same as in Common Lisp, but they support Markdown. Strings are UTF8 by default, and very fast. The community is very vibrant, open, and helpful. Simple questions get an answer within minutes, complicated ones (eg compiler internals) may take a bit longer, but are usually answered within a few hours or a day at most. If you are coming from the Common Lisp community, you will see quite a few familiar faces.

The library ecosystem already surpasses that of Common Lisp, at least for scientific programming. High-quality, well-tested code is available for linear algebra including sparse matrices (most of it in the standard library), optimization, differential equations, and automatic differentiation. The latter is simply amazing: by providing a type for dual numbers and a few operations, forward-mode AD can be used without any implementation overhead. Plotting libraries are available (mostly using foreign function backends), and R-like "dataframes" are under development.

Conclusion

Common Lisp has been and remains a great language, with many excellent features that preceded other languages by decades. It has an ANSI standard, which means that portable code written decades ago will run on a recent implementation. This is a great advantage, but at the same time this freezes the language development at the point the standard was finalized. No matter how flexible and forward-looking it is, it cannot predict and accommodate all possible advances for decades.

In contrast, Julia is rapidly evolving. At this stage, code that was written half a year ago is very likely to be broken with the most recent release.² The pace of change will most likely slow down a bit after 1.0 is released, but for now, expect a lot of churning.

On the other hand, programmers who used Common Lisp for scientific computing have always expected to get their hands dirty, since so little existing code was available. This is a good time to consider investing into Julia instead: you will get more done with less work, and you still get to program in a very elegant language that traces a lot of its roots to the Lisp family.

This is not the fastest, nor the most precise implementation, just a comparable example. ^[return]
An elegant deprecation mechanism is available, but that can't deal with some fundamental language changes. ^[return]

Clarification: on orphaning my Common Lisp libraries

2017-10-15T13:02:14+02:00

I have not been programming in Common Lisp for a few years, and since I find Julia a "much better Lisp", I am unlikely to go back to it in the foreseeable future. This is a clarification regarding some libraries I have written in Common Lisp and made public.

All of my Common Lisp libraries are orphaned

They are effectively abandonware as far as I am concerned. Since fixing issues and evaluating PRs entails a large fixed cost for which I don't have the time (setting up my CL environment again, understanding what I wrote years ago, thinking about code correctness and corner cases of the language spec that I have forgotten), I will ignore issues and pull requests. Sorry about this.
If you find anything of value in these libraries, please feel free to use that according to their licenses. You don't have to ask me explicitly.
If you want to take over maintaining any of these libraries, you don't have to ask me. Just fork, and start coding. If you have been consistently maintaining one of these libraries for a year or more, announce that you are resurrecting the library on the relevant Common Lisp forums. You can also drop me an e-mail and I will put a line in the README of my version that redirects users to your version. Eventually, you should convince Zach Beane to use your version in Quicklisp.
I cannot provide any significant help regarding the code due to time constraints. Some of it is documented, and most of it has unit tests, you have to figure out the rest yourself.

Which libraries are worth the effort?

let-plus is an extensible destructuring library. The syntax is versatile and intuitive.
LLA, aka Lisp Linear Algebra, is a wrapper for BLAS/LAPACK using native Common Lisp arrays. It is somewhat incomplete (eigenvalues need some work) but what is available works. It is fast on implementations which provide arrays for certain float element types, so that it does not have to copy the data, and is a bit slower on implementations that don't allow this. Still, copying is $O(n)$, while most LAPACK operations are $O(n^2)$ or worse, so this is not a huge concern. Nevertheless, it is possible that implementations that did not provide specialized arrays at the time I wrote LLA have caught up. You would need to extend the glue code to work with them.
cl-slice, array slices for native Common Lisp arrays.
cl-random, cl-num-utils, cl-rmath: random numbers, simple numerical algorithms, a wrapper for libRmath.
cl-colors, named colors and color combinations.

The rest are either early experiments, preliminary versions that evolved into the libraries above, or projects that did not pan out.

PS.: Some people asked why I switched to Julia from Common Lisp. A post about that will follow soon.

Branch prediction: yet another example

2017-10-04T10:12:02+02:00

Tomas Lycken linked a very nice discussion on StackOverflow about branch prediction as a comment on the previous post. It has an intuitive explanation (read it if you like good metaphors) and some Java benchmarks. I was curious about how it looks in Julia.

The exercise is to sum elements in a vector only if they are greater than or equal to 128.

function sumabove_if(x)
    s = zero(eltype(x))
    for elt in x
        if elt ≥ 128
            s += elt
        end
    end
    s
end

This calculation naturally has a branch in it, while the branchless version, using ifelse, does not:

function sumabove_ifelse(x)
    s = zero(eltype(x))
    for elt in x
        s += ifelse(elt ≥ 128, elt, zero(eltype(x)))
    end
    s
end

The actual example has something different: using tricky bit-twiddling to calculate the same value. I generally like to leave this sort of thing up to the compiler, because it is much, much better at it than I am, and I make mistakes all the time; worse, I don't know what I actually did when I reread the code 6 months later. But I included it here for comparison:

function sumabove_tricky(x::Vector{Int64})
    s = Int64(0)
    for elt in x
        s += ~((elt - 128) >> 63) & elt
    end
    s
end

Following the original example on StackOverflow, we sum 2^15 random integers in 1:256. For this, we don't need to worry about overflow. We also sum the sorted vector: this will facilitate branch predicion, since the various branches will be contiguous.

I also benchmark a simple version using generators:

sumabove_generator(x) = sum(y for y in x if y ≥ 128)

Benchmarks ($μ$s)
	random	sorted
`if`	139	28
`ifelse`	21	21
`if` & `sort`	96	n/a
tricky	27	27
generator	219	168

Benchmarks are in the table above. Note that

for the version with if, working on sorted vectors is dramatically faster (about 5x).
the non-branching ifelse version beats them hands down, and naturally it does not care about sorting.
if you have to use if, then you are better off sorting, even if you take the time of that into account.
generators are susprisingly bad.
the tricky bit-twiddling version is actually worse than ifelse (which reinforces my aversion to it).

Self-contained code for everything is available below.

download code as code.jl

CPU pipelines: when more is less

2017-10-03T11:40:58+02:00

I have been working on micro-optimizations for some simulation code, and was reminded of a counter-intuitive artifact of modern CPU architecture, which is worth a short post.

Consider (just for the sake of example) a very simple (if not particularly meaningful) function,

\[ f(x) = \begin{cases} (x+2)^2 & \text{if } x \ge 0,\\ 1-x & \text{otherwise} \end{cases} \]

with implementations

f1(x) = ifelse(x ≥ 0, abs2(x+2), 1-x)
f2(x) = x ≥ 0 ? abs2(x+2) : 1-x

f1 calculates both possibilities before choosing between them with ifelse, while f2 will only calculate values on demand. So, intuitively, it should be faster.

But it isn't...

julia> x = randn(1_000_000);

julia> using BenchmarkTools

julia> @btime f1.($x);
  664.228 μs (2 allocations: 7.63 MiB)

julia> @btime f2.($x);
  6.519 ms (2 allocations: 7.63 MiB)

...it is about 10x slower.

This can be understood as an artifact of the instruction pipeline: your x86 CPU likes to perform similar operations in staggered manner, and it does not like branches (jumps) because they break the flow.

Comparing the native code reveals that while f1 is jump-free, the if in f2 results in a jump (jae):

julia> @code_native f1(1.0)
        .text
Filename: REPL[2]
        pushq   %rbp
        movq    %rsp, %rbp
        movabsq $139862498743472, %rax  # imm = 0x7F34468E14B0
        movsd   (%rax), %xmm2           # xmm2 = mem[0],zero
Source line: 1
        addsd   %xmm0, %xmm2
        mulsd   %xmm2, %xmm2
        movabsq $139862498743480, %rax  # imm = 0x7F34468E14B8
        movsd   (%rax), %xmm3           # xmm3 = mem[0],zero
        subsd   %xmm0, %xmm3
        xorps   %xmm1, %xmm1
        cmpnlesd        %xmm0, %xmm1
        andpd   %xmm1, %xmm3
        andnpd  %xmm2, %xmm1
        orpd    %xmm3, %xmm1
        movapd  %xmm1, %xmm0
        popq    %rbp
        retq
        nopw    %cs:(%rax,%rax)

julia> @code_native f2(1.0)
        .text
Filename: REPL[3]
        pushq   %rbp
        movq    %rsp, %rbp
Source line: 1
        xorps   %xmm1, %xmm1
        ucomisd %xmm1, %xmm0
        jae     L37
        movabsq $139862498680736, %rax  # imm = 0x7F34468D1FA0
        movsd   (%rax), %xmm1           # xmm1 = mem[0],zero
        subsd   %xmm0, %xmm1
        movapd  %xmm1, %xmm0
        popq    %rbp
        retq
L37:
        movabsq $139862498680728, %rax  # imm = 0x7F34468D1F98
        addsd   (%rax), %xmm0
        mulsd   %xmm0, %xmm0
        popq    %rbp
        retq
        nopl    (%rax)

In my application the speed gain was more modest, but still sizeable. Benchmarking a non-branching version of your code is sometimes worth it, especially if it the change is simple and both branches of the conditional can be run error-free. If, for example, we had to calculate

g(x) = x ≥ 0 ? √(x+2) : 1-x

then we could not use ifelse without restricting the domain, since √(x+2) would fail whenever x < -2.

Julia Base contains many optimizations like this: for a particularly nice example see functions that use Base.null_safe_op.

Blog redesign 2.0

2017-09-29T19:55:44+02:00

I have redesigned my blog (again), mainly tweaking the CSS and hopefully achieving better support on small screens (which are still not ideal for math, but now you should get a red warning float at the bottom).

I also re-did the feed code so that it would render better on juliabloggers.com. Now the whole content of each post should be scraped seamlessly, and appear correctly. However, the only way to test the whole toolchain is to do it live, and I apologize if something is still not perfect and you get bogus updates (BTW, this post should not show up in the Julia feed).

Highlights of the changes:

responsive design,
nicer fonts,
line breaks in MathJax when necessary and supported,
better highlighting (Julia now looks especially nice),
embedded code blocks with a download link,
better image placement,
Emacs screenshots now use a branch of emacs-htmlize, which hopefully gets merged soon.

As always, the source code for the whole site is available.

Automatic differentiation of discontinuous integrals

2017-09-15T16:03:10+02:00

This is a much simplified writeup of a problem I encountered, in a self-contained blog post.

You want to approximate the integral

\[ I(\theta) = \int 1(g(x) > 0) f(x,\theta) dx \]

where $g$ is a continuous function, and $f(x,\theta)$ is a parametric distribution over $x$. Everthing is continous, and thus $I$ is, too.

You face the following constraints:

$g$ is a black box. We will pretend that you can't invert it (except for checking our results, of course).
You can calculate the probability density function $f$ and even draw $x$'s for a particular $\theta$, but that's pretty much it. You don't even get a cdf! (Again, except for checking our results.)

Using Monte Carlo methods, you can do the following:

draw $x_i \sim F(\cdot, \theta)$, $i=1,\dots,N$,
approximate

\[ I(\theta) \approx \frac{1}{N}\sum_i 1(g(x_i) > 0) \]

You could code this in Julia as

d = distr(θ)   # suppose this returns some distribution that supports Base.rand
x = rand(d, N)
mean(g.(x) > 0)

So far, neat and simple. Now, the fly in the ointment: you need the derivative $I'(\theta)$ for optimization or Hamiltonian Monte Carlo. The problem is that you cannot ForwardDiff your way through the code above: AD'ing a discontinuous step function will just give you $0$, and rand does not work with ForwardDiff.Duals anyway (which is very sensible).

However, there is a solution: rewrite the approximation as

\[ I(\theta; \theta_0) \approx \frac{1}{N}\sum_i 1(g(x_i) > 0) \frac{f(x_i,\theta)}{f(x_i,\theta_0)} \]

where $\theta_0$ is the parameter used to simulate the $x_i$. Differentiate the above at $\theta = \theta_0$. This approximates

\[ I'(\theta) = \int 1(g(x) > 0) \frac{\partial f(x,\theta)/\partial \theta}{f(x,\theta)}f(x,\theta) dx = \int 1(g(x) > 0) \partial f(x,\theta)/\partial \theta dx \]

which is exactly what you want.

Assume that the actual calculation is very complicated, so we would rather avoid implementing it for the integral and the derivative separately. It turns out that this is very simple to do with ForwardDiff.Dual values: the code is literally a one-liner and a fallback method:

elasticity(x::Real) = one(x)
elasticity(x::Dual{T,V,N}) where {T,V,N} = Dual{T}(one(V), partials(x) / value(x))

which you can use in a function like

integral_approx(g, d, x) = mean((g.(x) .> 0) .* elasticity.(pdf.(d, x)))

I demonstrate this with $g(x) = x$ and $x \sim \text{Normal}(\theta, 1)$, for which of course we know that the analytical values for $I$ and $I'$ are the right tail probability and the pdf at 0, respectively.

Graphs below show that the approximation is reasonable — we could make it much better with low-discrepancy sequences, but that is an orthogonal issue.

It is amazing how much you can accomplish with two lines of code in Julia! The problem that motivated this blog post is multivariate with irregular regions over which $\{ x: g(x) > 0 \}$, but I used elasticity as above.

Self-contained code for everything is available below.

download as code.jl

using ForwardDiff
import ForwardDiff: Dual, value, partials, Tag
using Distributions
using Plots
using LaTeXStrings
gr()

######################################################################
# elasticity calculation
######################################################################

"""
    elasticity(x)

Calculate `x/x`, stripping `x` of partial derivative information. Useful for
calculating elasticities of the form `∂f/f` using ForwardDiff.
"""
elasticity(x::Real) = one(x)

elasticity(x::Dual{T,V,N}) where {T,V,N} = Dual{T}(one(V), partials(x) / value(x))

######################################################################
# example application
######################################################################

integral_approx(g, d, x) = mean((g.(x) .> 0) .* elasticity.(pdf.(d, x)))

"Helper function that returns the value and derivative at the same time."
function value_and_derivative(f::F, x::R) where {F,R<:Real}
    T = typeof(Tag(F, R))
    y = f(Dual{T}(x, one(x)))
    value(y), partials(y, 1)
end

distr(θ) = Normal(θ, 1)

g(x) = x

function ID_analytical(θ)
    d = distr(θ)
    ccdf(d, 0), pdf(d, 0)
end

function ID_approx(θ)
    x = rand(distr(θ), 1000)
    value_and_derivative(θ->integral_approx(g, distr(θ), x), θ)
end

θs = linspace(-2, 2, 51)
ID0s = ID_analytical.(θs)
ID1s = ID_approx.(θs)

scatter(θs, first.(ID0s), xlab = "\\theta", ylab = "integral",
        label = "analytical", legend = :topleft)
scatter!(θs, first.(ID1s), label = "approximation")
savefig("integral.svg")

scatter(θs, last.(ID0s), xlab = "\\theta", ylab = "derivative",
        label = "analytical", legend = :topleft)
scatter!(θs, last.(ID1s), label = "approximation")
savefig("derivative.svg")

log1p in Julia

2017-09-13T11:18:59+02:00

edit: fixed bogus interaction of MathJax and code highlighting.

edit2: added benchmarks.

This is a follow-up on a question I asked on the Julia forums about calculating

\[ \text{log1p}(x) = \log(1+x) \]

This calculation is tricky because if $x \approx 0$,

\[ \log(1+x) \approx x \]

while as $x \to \infty$, $\log(1+x)$ approaches $\log(x)$, so simply using log(1+x) will not be as accurate as it could be. Numerical analysts have developed specialized methods for these calculations that try to get an accurate answer, and all programming languages serious about numerical calculations have an implementation either in the core language or a library.

Julia's Base.log1p currently suggests that Base.Math.JuliaLibm.log1p would be preferable, but then I was wondering why isn't that the default? So I decided to perform a trivial numerical experiment, calculating the error for both, and also benchmark the two methods.

Accuracy

The key question is what to compare the results with. One could compare to an existing "gold standard" implementation, or simply calculate the results using a higher precision representation. Fortunately, Julia has BigFloats available right out of the box.

The graph below shows the base-2 logarithm of the relative error for Base.log1p vs $\log\_2(1+x)$; horizontal lines are log2(eps()) and log2(eps())+1. This suggests that Base.log1p is very accurate, but not as good as it could be when $x \approx 0$.

The next plot shows the relative accuracy of the relative error above, comparing Base.Math.JuliaLibm.log1p to Base.log1p (lower values better). In these simulations, Base.Math.JuliaLibm.log1p is never worse, but sometimes significantly better (resulting in an extra binary digit of accuracy). This matters especially when $x \approx 0$.

The next plot confirms this.

Speed

I also evaluated relative speed. The graph below shows the relative runtimes, obtained using BenchmarkTools.@belapsed. Values below $1$ mean that Base.Math.JuliaLibm.log1p is faster: indeed, this seems to be the case except for values very close to $0$, where there is a 10–20% performance penalty. At other values, Base.Math.JuliaLibm.log1p is 30–40% faster.

Conclusion

For values near $0$, Base.Math.JuliaLibm.log1p is more accurate, at a slight performance cost.
For values away from $0$, it is at least as accurate as Base.log1p, and 30—40% faster.

To me, this suggests that Base.Math.JuliaLibm.log1p should be the default method — mostly because the extra accuracy is more important to me than the slight performance cost.

Code is available below.

download as code.jl

# consistent random numbers
srand(UInt32[0xfd909253, 0x7859c364, 0x7cd42419, 0x4c06a3b6])

"""
    err(x, [prec])

Return two values, which are the log2 relative errors for calculating
`log1p(x)`, using `Base.log1p` and `Base.Math.JuliaLibm.log1p`.

The errors are calculated by compating to `BigFloat` calculations with the given
precision `prec`.
"""
function err(x, prec = 1024)
    yb = log(1+BigFloat(x, prec))
    e2(y) = Float64(log2(abs(y-yb)/abs(yb)))
    e2(log1p(x)), e2(Base.Math.JuliaLibm.log1p(x))
end

z = exp.(randn(20000)*10)       # z > 0, Lognormal(0, 10)
x = z .- 1                      # x > -1
es = map(err, x)                # errors

using Plots; gr()               # plots

scatter(log2.(z), first.(es), xlab = "log2(x+1)", ylab = "log2 error of Base.log1p",
        legend = false)
hline!(log2(eps())-[0,1])
Plots.svg("Base_log1p_error.svg")
scatter(log2.(z), last.(es) .- first.(es), xlab = "log2(x+1)",
        ylab = "improvement (Base.Math.JuliaLibm.log1p)", legend = false)
Plots.svg("JuliaLibm_improvement.svg")
scatter(log2.(z), last.(es), xlab = "log2(x+1)",
        ylab = "log2 error of Base.Math.JuliaLibm.log1p", legend = false)
hline!(log2(eps())-[0,1])
Plots.svg("JuliaLibm_log1p_error.svg")

######################################################################
# WARNING: these run for a very long time
######################################################################
using BenchmarkTools

z = exp.(vcat(randn(200)*10, rand(200)*0.1)) # z > 0, more values around 
x = z .- 1                                   # x > -1
b1 = [@belapsed log1p($x) for x in x]        # WARNING: takes forever
b2 = [@belapsed Base.Math.JuliaLibm.log1p($x) for x in x] # ditto

scatter(log2.(z), b2 ./ b1, xlab = "log2(x+1)",
        ylab = "time Math.JuliaLibm.log1p / log1p", legend = false, yticks = 0:0.2:1.2)
hline!([1])
Plots.svg("relative_time.svg")

Emacs customizations for julia-mode

2017-08-28T14:41:58+02:00

I find the following customizations very useful for editing Julia code in Emacs. Add them to julia-mode-hook, eg

(defun customize-julia-mode ()
  "Customize julia-mode."
  (interactive)
  ;; my customizations go here
  )

(add-hook 'julia-mode-hook 'customize-julia-mode)

Highlight FIXME/TODO/…

When I just want to note something in a comment for future reference, I prefer to have certain words highlighted. You can use something like this:

(font-lock-add-keywords nil
                        '(("\\<\\(FIXME\\|TODO\\|QUESTION\\|NOTE\\)"
                        1 font-lock-warning-face t)))

This is what it looks like:

            chklapackerror(info[])
            if any(ifail .!= 0)
                # TODO: better error message / type
                error("failed to converge eigenvectors:\n$(nonzeros(ifail))")
            end

Highlight symbols

After

(require 'highlight-symbol)

add a hook for

(local-set-key [(control ?c) ?s] 'highlight-symbol-at-point)
(local-set-key [(control ?c) ?n] 'highlight-symbol-next)
(local-set-key [(control ?c) ?p] 'highlight-symbol-prev)

This highlights symbols with C-c s:

function issymmetric(A::AbstractMatrix)
    indsm, indsn = indices(A)
    if indsm != indsn
        return false
    end
    for i = first(indsn):last(indsn), j = (i):last(indsn)
        if A[i,j] != transpose(A[j,i])
            return false
        end
    end
    return true
end

Fill docstrings

This is useful if you want to use M-q on docstrings.

(defun julia-fill-string ()
  "Fill a docstring, preserving newlines before and after triple quotation marks."
  (interactive)
  (if (and transient-mark-mode mark-active)
      (fill-region (region-beginning) (region-end) nil t)
    (cl-flet ((fill-if-string ()
                              (when (or (looking-at (rx "\"\"\""
                                                        (group
                                                         (*? (or (not (any "\\"))
                                                                 (seq "\\" anything))))
                                                        "\"\"\""))
                                        (looking-at (rx "\""
                                                        (group
                                                         (*? (or (not (any "\\"))
                                                                 (seq "\\" anything))))
                                                        "\"")))
                                (let ((start (match-beginning 1))
                                      (end (match-end 1)))
                                  ;; (ess-blink-region start end)
                                  (fill-region start end nil nil nil)))))
      (save-excursion
        (let ((s (syntax-ppss)))
          (when (fourth s) (goto-char (ninth s))))
        (fill-if-string)))))

Add

(local-set-key (kbd "M-q") 'julia-fill-string)

to the mode hook.

Highlight things after column 80

I add this to the mode hook:

(set-fill-column 80)

I also use whitespace globally:

(require 'whitespace)
(setq whitespace-style '(face empty tabs lines-tail trailing))
(global-whitespace-mode t)

This is what it looks like:

    QR{T,S}(factors::AbstractMatrix{T}, τ::Vector{T}) where {T,S<:AbstractMatrix} = new(factors, τ)
end
QR(factors::AbstractMatrix{T}, τ::Vector{T}) where {T} = QR{T,typeof(factors)}(factors, τ)

Hungry delete-mode

Add this to the mode hook:

(hungry-delete-mode)

and then backspace and delete will remove all whitespace near the point in the relevant direction.

In case you are wondering, the theme is alect-light.

Local packages in a separate directory in Julia

2017-08-23T10:10:48+02:00

I run Pkg.update() fairly often to stay up to date and benefit from the latest improvements of various packages. I rarely ever pin to a specific package version, but I occasionally checkout master for some packages, especially if I am contributing.

Despite updating regularly, I found that the documentation subtly diverged from what I was experiencing for some packages. After looking into the issue, I learned that I was 2–3 minor versions behind despite updating regularly. For example, when I would

Pkg.update("ForwardDiff")

I would be told that there is a new version, and to get it I should update ReverseDiff and Optim. But

Pkg.update("ForwardDiff", "ReverseDiff", "Optim")

would just run quietly without updating.

I could not figure out the cause for this and did not want to get sidetracked debugging it, so I decided to wipe the package directory and start over. However, in order to do this, I had to make sure that no code is lost, especially for local packages. First, I moved my local packages into a separate directory, and added that to LOAD_PATH in .juliarc.jl:

push!(LOAD_PATH, expanduser("~/src/julia-local-packages/"))

Then I ran multi-git-status to make sure that there were no unpushed changes. Finally, I deleted the package directory and reinstalled everything. Surprisingly, Pkg.add ran much faster than before.

In case I have to do this again, I decided to keep my local packages separate — the only drawback is that Pkg.test now can’t find them. A workaround is below, using some code from Base.Pkg:

"""
    local_test(pkgname, [coverage])

Find and test a package in `LOAD_PATH`. Useful when the package is outside
`Pkg.dir()`.

Assumes the usual directory structure: package has the same name as the module,
the main file is in `src/Pkgname.jl`, while tests are in `test/runtests.jl`.
"""
function local_test(pkgname; coverage::Bool=false)
    module_path = Base.find_in_path(pkgname, nothing)
    src_dir, module_file = splitdir(module_path)
    dir = normpath(src_dir, "..")
    test_path = joinpath(dir, "test", "runtests.jl")
    @assert isfile(test_path) "Could not find $(test_path)"
    Base.cd(dir) do
        try
            color = Base.have_color? "--color=yes" : "--color=no"
            codecov = coverage? ["--code-coverage=user"] : ["--code-coverage=none"]
            compilecache = "--compilecache=" * (Bool(Base.JLOptions().use_compilecache) ? "yes" : "no")
            julia_exe = Base.julia_cmd()
            run(`$julia_exe --check-bounds=yes $codecov $color $compilecache $test_path`)
            info("$module_file tests passed")
        catch err
            Base.Pkg.Entry.warnbanner(err, label="[ ERROR: $module_file ]")
        end
    end
end

Compared to simply include("wherever-it-is/runtests.jl"), this has the advantage of running a separate Julia process, so your workspace does not contaminate the test environment and in case of segfaults, the parent process won’t be affected.

Hopefully, the code above will be obsolete once Pkg3 is released, but until then it is a useful workaround.

edit: function above was corrupted during copy-paste, corrected.

Emacs 25.2 on Ubuntu

2017-07-03T10:49:56+02:00

Emacs is undoubtedly the most important program on my computers. On my laptop, I use it to keep track of stuff with org-mode, read mail with mu4e, edit LaTeX with AUCTeX, and of course program. On servers, the first alias I define is usually qe='emacs -Q -nw', which give me a fast and responsive editor. With helm, doing just about anything (eg locating files, rgreping for something) is orders of magnitude faster and more convenient than any alternative I have tried.

I also try to keep up with the latest versions for software in general. Usually, whatever Ubuntu stable/Debian testing has is good enough not to justify the extra effort, but when I really need it, I grab the source and compile. That is usually only a minor hassle, but I try to restrict it to a few critical programs, otherwise it adds up. The major issue is not compiling, but having cruft in the filesystem (despite stow and checkinstall, it piles up). So I try to avoid it if I can.

Emacs 25.2 was released in April 2017, but there is no sign of an Ubuntu package for it yet. On various forums the PPA of kelleyk is recommended, but that does not have 25.2 for 17.04 (some files clash if you install previous versions).

Fortunately, Robert Bruce Park has now added Emacs 25.2 to the Ubuntu Emacs Lisp PPA, so having the latest of your favorite editor is only an add-apt-repository away. You may want to add a file to /etc/apt/preferences.d with contents

Package: *
Pin: release o=LP-PPA-ubuntu-elisp
Pin-Priority: 600

to make sure the right packages are installed.

Sampling variation in effective sample size estimates (MCMC)

2017-06-12T16:25:57+02:00

Introduction

MCMC samples, used in Bayesian statistics, are not independent --- in fact, unless one uses specialized methods or modern HMC, posterior draws are usually at highly autocorrelated. For independent draws,

\[ \text{variance of simulation mean} \propto \frac1N \]

where $N$ is the sample size, but for correlated draws, one has to scale the sample size with a factor

\[ \tau = \frac{1}{1+2\sum_{k=1}^\infty \rho_k} \]

where $\rho_k$ is the lag-$k$ autocorrelation. $\tau N$ is the effective sample size.

Usually, $\rho_k$ is estimated from the data using the variogram

\[ V_k = \frac{1}{N-k} \sum_{i=1}^{N-k} x_i x_{i+k} \]

from which we obtain

\[ \rho_k = 1-\frac{V_k}{2\text{var}(x)} \]

where an estimator for the variance is also used. Then, to avoid using noisy estimates, we only add up to the last $K$ where

\[ \rho_{K} + \rho_{K+1} \ge 0 \]

I will call $K$ the last lag. Stan does something slightly different, using FFT for autocorrelations, and cutting off at the first negative $\rho_K$, but for HMC this does not make a whole lot of difference.

The sampling variation

I was coding up the above calculation, and needed some unit tests. Surprisignly, I could not find anything on the sampling variation of $\tau$, so I wrote some simulations in Julia (source code for everything). I did the following simulation exercise:

for a given autocorrelation coefficient $\phi$, simulate $N$ draws from the AR(1) process $ x_t = \phi x_{t-1} + \sigma \epsilon_t \qquad \epsilon_t \sim \text{Normal}(0,1), \text{IID} $
calculate $\tau$ and $K$,
repeat 1000 times and plot the results.

I use $N=1000$ and $N=10000$, as these would be typical sample sizes, first for a fairly efficient algorithm, then for a more stubborn but still manageable posterior.

IID samples

Let $\phi=0$, then we expect $\tau=1$ (red line in histogram, coefficient of variation on top).

Results with $\phi=0$ (IID), $N=1000$. (a) $\tau$, (b) last lag $K$, (c) scatterplot.

Results with $\phi=0$ (IID), $N=10000$. (a) $\tau$, (b) last lag $K$, (c) scatterplot.

With $1000$ samples, there is a lot of variation in ESS: 800 could show up very easily in practice. $600$ is not improbable either. Using up to $10$ lags is not uncommon. For $10000$ samples, the precision is improved considerably, we commonly use $2$ or $4$ lags. For both sample sizes, notice the high correlation between the last lag $K$, and $\tau$: given the method above, using more lags increases $\tau^{-1}$, so this is to be expected.

AR(1) samples with $\rho=0.5$

This is a more autocorrelated process, here theory tells us that $\tau$=1/3.

Results with $\phi=0.5$, $N=1000$. (a) $\tau$, (b) last lag $K$, (c) scatterplot.

Results with $\phi=0.5$, $N=10000$. (a) $\tau$, (b) last lag $K$, (c) scatterplot.

Notice that $\tau$ is now more dispersed, compared to the IID case. Even with 10000 samples, the coefficient of variation is 6%, with 1000 it is around 1/6. In practice, expect effective sample sizes all over the place.

AR(1) samples with $\rho=0.8$

This is an even more autocorrelated process, here theory tells us that $\tau$=1/9.

Results with $\phi=0.8$, $N=1000$. (a) $\tau$, (b) last lag $K$, (c) scatterplot.

Results with $\phi=0.8$, $N=10000$. (a) $\tau$, (b) last lag $K$, (c) scatterplot.

There is now so much variation that in order to get an estimate for ESS that we can use for comparing various MCMC implementations, we need to run much more than $1000$ samples.

Conclusion

For unit testing ESS calculations, I will need to use 10000 samples, with $\pm10$ or similar error bands.
As a rule of thumb, I would ignore less than 1.5x variation in ESS for 1000 samples, or run longer chains: it may be just random noise.

Bibliography

Gelman, Andrew, et al. 2013. Bayesian data analysis. 3rd edition. Chapman & Hall/CRC.
Stan Development Team. 2016. Stan Modeling Language Users Guide and Reference Manual, Version 2.15.0. http://mc-stan.org

Blog redesign

2017-06-12T14:12:43+02:00

I am in the process of rebuilding my personal website using Hugo. I tried various themes, including hugo-academic, but in the process of adapting them to my needs I realized that it is less work to write one from scratch.

The result is now 80% ready (blog works, automatic listing of research papers will take some more work), and the source is on Github. It is available under the CC-BY-SA license, feel free to adapt parts from it, not that there is anything special in there.

Hugo is really an excellent framework, it is clean, logical, and allows a lot of code reuse. I ~~wasted~~ spent most of the time on fiddling with CSS, and I am still not 100% satisfied with the result, but at some point I decided to stop. SCSS was extremely useful for writing organized CSS.

I am especially satisfied with moving the code highlighting to the generator side. The only non-static parts are now Disqus and MathJax.

Research

2017-06-11T14:48:06+02:00

Working papers

Couples’ times use and aggregate outcomes: evidence from a structural model. Joint with Almut Balleer and Monika Merz.
The structure of labor market flows. PDF
Accounting for Idiosyncratic Wage Risk Over the Business Cycle. Joint with Alisdair McKay. PDF

Short notes

Consistent local approximation in continuous time. PDF

Published papers

Estimating Linearized Heterogeneous Agent Models Using Panel Data. Joint with Michael Reiter. JEDC 2020, also formatted for screen reading.
Frictional wage dispersion with Bertrand competition: an assessment. RED 2013.

Getting a nice += in LaTeX math

2017-05-24T12:59:53+02:00

I am working on an appendix for a paper that uses MCMC, and I decided to document some change of varible calculations in the interest of reproducibility (they are quite complex, because of multivariate determinants). But how can I typeset them nicely in $\LaTeX$?

\mathtt{target} += J_f

gives $$ \mathtt{target} += J_f $$ which is to be expected, as + is a binary operator and = is a relation, so $\LaTeX$ is not expecting them to show up this way.

We can remedy this as

\mathtt{target} \mathrel{+}= J_f

which shows up as $$ \mathtt{target} \mathrel{+}= J_f $$ which is an improvement, but is still not visually appealing.

Making the + a bit smaller with

\mathrel{\raisebox{0.19ex}{$\scriptstyle+$}}=}

yields $$ \mathtt{target} \mathrel{\raise{0.19ex}{\scriptstyle+}} = J_f $$ which looks OK enough to preclude further tweaking. Note that MathJax does not support \raisebox, but you can use

\mathrel{\raise{0.19ex}{\scriptstyle+}} = J_f

which renders the as above.

Two tricks for change of variables in MCMC

2017-05-23T16:39:26+02:00

Change of variables are sometimes advantageous, and occasionally inevitable for MCMC if you want efficient sampling, or to model a distribution that was obtained by a transformation. A classic example is the lognormal distribution: when

\[\log(y) \sim N(\mu, \sigma^2)\]

one has to adjust the log posterior by $-\log y$ since

\[\frac{\partial \log(y)}{\partial y} = \frac{1}{y}\]

and

\[\log(1/y) = -\log(y).\]

In Stan, one would accomplish this as

target += -log(y)

In general, when you transform using a multivariate function $f$, you would adjust by

\[\log\det J_f(y)\]

which is the log of the determinant of the Jacobian — some texts simply refer to this as "the Jacobian".

The above is well-known, but the following two tricks are worth mentioning.

Chaining transformations

Suppose that you are changing a variable by using a chain of two functions $f \circ g$. Then

\[ \log\det J_{f \circ g}(y) = \log \bigl(\det J_f(g(y)) \cdot \det J_g(y)\bigr) \\\\ = \log\det J_f(g(y)) + \log\det J_g(y) \]

which means that you can simply add (the log determinant of) the Jacobians, of course evaluated at the appropriate points.

This is very useful when $f \circ g$ is complicated and $J_{f\circ g}$ is tedious to derive, or if you want to use multiple $f$s or $g$s and economize on the algebra. From the above, it is also easy to see that this generalizes to arbitrarily long chains of functions $f_1 \circ f_2 \circ \dots$.

This trick turned out to be very useful when I was fitting a model where a transformation was general to both equilibrium concepts I was using (a noncooperative game and a social planner), so I could save on code. Of course, since #2224 is WIP, I had to copy-paste the code, but still saved quite a bit of work.

Transforming a subset of variables

Suppose $x \in \mathbb{R}^m$ and $y \in \mathbb{R}^n$ are vectors, and you are interested in transforming to

\[ z = f(x,y) \]

where $x$ and $z$ have the same dimension. It is useful to think about this transformation as

\[ g(x,y) = [f(x,y), y]^\top \]

where $g : \mathbb{R}^{m+n} \to \mathbb{R}^{m+n}$. Since $y$ is mapped to itself,

\[ J_g = \begin{bmatrix} J_{f,x} & J_{f,y} \\\\ 0 & I \end{bmatrix} \]

has a block structure, where

\[ J_{f,x} = \frac{\partial f(x,y)}{\partial x} \]

and similarly for $J_{f,y}$. For the calculation of the determinant, you can safely ignore the latter, and $\log \det I = 0$, so

\[ \log\det J_g = \log\det J_{f,x} \]

Disabling Privacy Badger

2017-05-02T16:35:37+02:00

Firefox has been my primary browser for the last decade. I find it very fast and convenient, and use quite a few addons, including Google Scholar Button, NoScript, and uBlock Origin.

I can’t recall exactly when, but about half a year ago various sites started to break. I have not bothered to debug what’s happening, suspecting it was was a combination of various plugins, but continued to use Firefox and installed openwith to open broken pages in Chromium (yes, I know, the ultimate kludge). More sites became broken, and I found that now I am spending 99% of my time in Chromium, which I don’t like that much, but moreover, it is a resource hog: while using the plugins above I can get the CPU load of Firefox to around 1–2% when I am not using it, Chromium drains my laptop battery in effectively half the time. To make things worse, Chromium apparently can’t open links in the background like Firefox, and insists on raising the window every time I open a link from another process, which is distracting.

Finally, Julia’s Discourse forum started showing up empty, which was the last straw and I went through my plugins. It turns out hat Privacy Badger was responsible for everything: apparently it relies on heavy-handed heuristics and breaks a lot of webpages. One can report this, but instead of reporting around 10–20 pages which were broken, I simply removed the plugin. I have a lot of respect for what the Electronic Frontier Foundation does, but I am not sure that this plugin is very useful.

So finally I got back the web the way I like it. Moral of the story: temporary workarounds become permanent, and bite back.

Blogging with Hugo, Julia, Weave.jl

2017-03-30T10:02:32+02:00

I have made a PR to Weave.jl which Matti Pastell kindly merged recently. This allows a relatively smooth workflow for blogging using the static website generator Hugo, and generating some pages with plots and evaluated Julia results. I made the source for my blog available so that others can use it for their own blogging about Julia. An example is this post.

The gist of the workflow is as follows:

for posts which do not need Weave, just use Hugo. Make sure you read their excellent tutorial. This very fast.
for posts which contain Julia code and generated plots, use a script to generate a skeleton file in a separate directory, and work on that. Call another script to generate the .md file using Weave.jl. This is the slow part, so it is not automated.

The README gives more details. Feel free to ask questions here. If you have a better workflow, I would like to hear about it.

Teaching a course using Julia

2017-03-24T14:34:22+01:00

I just finished teaching a graduate course on practical aspects of dynamic optimization to our economics students. This year, for the first time, I taught the course using Julia, and this is a writeup of the experience. This was an intensive, 10-week course, with two classes per week, taught in the computer lab. A course on the theory of dynamic optimization was a prerequisite, this one was all about the actual numerical methods. The students had prior exposure to R and Matlab, and some of them have been using Julia for a while. In some classes, I talked about theory, sometimes I wrote code, ran it, and made improved versions, sometimes we treated the class as a practice session.

I wanted to focus on the actual methods, so I decided to use a subset of the language, “Julia light”, using the following concepts:

scalars, algebra, arrays, indexing
functions (with very basic dispatch, on an argument that contained problem parameters)
control flow: for and if
comprehension, concatenation
structures (only immutable)
docstrings

The purpose of the course was to show that one can easily implement seemingly abstract methods encountered in textbooks, dissect them, look at the caveats, and possibly adapt them to particular problems. Writing what I think of as production code would have involved teaching many new concepts to a class coming with very heterogeneous experience in programming, so I decided to steer clear of the following topics:

modules
macros
the type system
writing efficient code (even though we ended up doing a bit on that, and benchmarking, could not resist)

We used NLsolve, Optim, and Plots extensively, and ForwardDiff under the hood. Parameters was very useful for clean code.

Perspective of the instructor

Even when I wrote something in a suboptimal manner, it turned out to be fast enough. Julia is great in that respect. However, compilation time dominated almost everything that we did, especially for plots.

I was using Jupyter notebooks, inspired by 18.S096. While I am much, much slower writing code in Jupyter compared to Emacs, I think that this turned out to be a benefit: jumping around between windows is very difficult to follow. Interact was just great.

I made a small package for code we wrote in class, and distributed new code via Pkg.update(). This worked well most of the time.

We were using v0.5.0 and later transitioned to v0.5.1, which was seamless.

Since I was not using modules, sometimes the best way to extricate myself from a state was restarting the kernel. This became a running joke among the students (“when in doubt, restart the kernel”). This actually worked very well for the infamous #265.

Jupyter is great for interactive notes in class. I mixed a great deal of marked up text and LaTeX equations into the workbooks. Problem sets were handed in using Jupyter notebooks, and the exam (solving a dynamic programming problem) was also written in and graded as a notebook.

Unicode specials are addictive. Once you learn about α, you never name a variable alpha again.

Perspective of the students

I talked to the class at the end of the course about their experience with Julia, and some of them individually. The biggest issue for them was lack of easily searchable answers to common questions: for R and Matlab, a “how do you …” query turns up 100+ answers because many people have encountered the problem before. This was not the case for Julia. Lack of examples was an especially difficult issue for plots.

Debugging in Jupyter was difficult, since it mostly amounted to debugging by bisection, isolation, and occasionally printing. Students found some of the error messages cryptic (especially when it was about not having a matching method, since we did not really go into the type system).

The most puzzling transient bugs were because of #265 (“but I recompiled that function!"). This was solved by restarting the kernel, so the latter became somewhat of a panacea. Since compilation time dominated everything, this slowed things down considerably.

Takeaway

Would definitely do it again. Even with the issues, Julia was the most comfortable language to teach in.

Julia + Weave.jl + hugo test

2017-03-03T13:11:53+01:00

Testing the Hugo formatter for Weave.jl.

Testing inline code: 1+1=2.

Testing math: $$x^2+y^2 = \int_0^1 f(z) dz$$

Testing code:

1+1

Testing proper highlighting:

function foo(x, y)
    x+y
end

A plot:

x = 1:10
y = x.^2
scatter(x, y, legend = false)