Matthew Strawbridge's weblog

Serving Python on AWS Lambda with Zappa

Sat, 26 Apr 2025 13:00:00 +0100

This post introduces the Zappa project – a tool for deploying Python services to run on AWS Lambda, without needing a server running full-time. If you write Python applications and have access to an AWS environment, Zappa offers a distinctive deployment model that can be both cost-effective and scalable. It’s not right for every project, but when it fits, it can be an elegant solution worth knowing about.

What is Zappa?

Zappa is a fascinating library for running Python code, including web apps, on AWS Lambda.

So, instead of a web server running 24/7, your services get spun up on demand in response to HTTP requests as they come in, and then destroyed (or at least, allowed to sleep) again.

This can be a good fit in a couple of situations:

Low-traffic services. Because you pay only for the time when your service is actively executing code, sites with low traffic can be very inexpensive (say, less that $1/month and potentially free) to run in this way.
High-traffic services. Responding to web requests using serverless Lambda functions effectively gives you infinite parallel web servers.

Zappa does some really clever things behind the scenes to package up your Python code so that it runs smoothly on AWS Lambda (and it does need to be AWS – no other cloud services are supported so far). It builds a zip file containing all the dependencies needed by your Python code, automatically fetching Lambda-compatible wheels.

For deployment, Zappa builds the package Zip, uploads it to AWS S3 blob storage, loads it into a Lambda function, and deletes the S3 blob again.

Okay, but this only works for simple things, right?

Well, no. Zappa supports WSGI web servers, including full-blown Django applications.

For the project I decided to try Zappa with, I chose to use another library I was interested in learning about: nanodjango. This provides a simple, lightweight way to start a Django project using a single file, with the ability to convert it to a full-blown Django project automatically later.

Setting up Zappa in practice

This isn’t intended to be a full how-to. But I did want to provide some specific notes from my implementation about several specific areas: package zip sizes, AWS permissions, and keeping the service warm.

Package zip sizes

You can run zappa package [env] to create the package’s zip file locally. Then you can look inside it to see what’s included. Then you can specify exclude and exclude_glob entries in your zappa_settings.json file to trim the size down (and to ensure no sensitive information is included).

I used these settings:

        "exclude": [
            ".gitignore",
            ".pytest_cache",
            "README.md",
            "__pycache__",
            "_pytest",
            "black",
            "justfile",
            "mypy_extensions.py",
            "poetry",
            "poetry_plugin_export",
            "poetry.lock",
            "pyproject.toml",
            "pytest",
            "zappa_settings.json",
            "zappa_settings.py"
        ],
        "exclude_glob": [
            "*.dist-info",
            "big-day-diaries-prod-template*.json",
            "django/test/*",
            "docs/*",
            "test_*.py"
        ]

which took my package zip file size down from about 65 MiB to about 53 MiB – a reasonable reduction. I expect there’s more that could be pruned, particularly unused things inside the django/ directory.

Note that this is, shall we say, chunky for a Lambda function. Zappa issues a warning that going above 50 MiB probably won’t work, but it does (it runs fine, but AWS won’t allow you to edit it interactively).

Minimal AWS permissions set

In general, spinning up a working nanodjango site on AWS Lambda using Zappa was straightforward. But by far the most annoying part of the process was figuring out the minimal set of AWS permissions that were necessary to enable the deployment steps to work cleanly.

It’s good practice not to over-provision the permissions: just use the minimal subset that will work. Unfortunately, there’s no guidance in Zappa’s documentation about what’s required, so I fell back into trial and error:

try to deploy
get an error message
add the missing permission in AWS
tear down and try again.

I applied the following two AWS-managed policies:

AWSLambda_FullAccess
AWSCodeDeployRoleForLambda

In addition, I created a custom policy to allow access to the specific resources used in this deployment: the relevant Lambda execution role, S3 bucket, API Gateway routes, and CloudFormation stack. It also included limited permissions for IAM role creation and for interacting with associated services like CloudFront, Route 53, and CloudWatch Events.

I’m not sharing the full policy document here, partly because it’s specific to my setup and partly because I don’t want to mislead anyone into copying a configuration that might not be appropriate for their environment.

So, be prepared for some annoyance with this part of the process.

Keeping warm

AWS Lambda caches the Lambda functions for a few minutes so that they are available straight away when called multiple times in succession. After that, they are unloaded, and reloading them can take a significant amount of time (say 30 seconds for these large zip packages). For some workloads, that scale of delay would be fine, but for responding to Web requests it is too long.

Zappa includes a keep_warm setting, which will ping the deployed service every few minutes to keep it warm and in the cache. This way, the service is always available and can respond immediately to incoming requests. These ping calls are quick and cheap, so their cost is negligible.

Example service

To test all this out, I created a simple service for generating a custom PDF countdown calendar for a big day: a birthday, anniversary, wedding, etc.

After a few rounds of redesign, the single-page user interface looks like this:

Django is overkill for this: it’s essentially just a form for entering the title and date, validation of the form, generation of a PDF (I used ReportLab for this) and serving up the PDF. There’s no database to speak of in this case, and I was also able to directly embed the small images, which kept things nice and simple.

Here is an example extract of the PDFs that this service can generate:

Notice that it marks public holidays and draws quite a pleasing red swirl around the big day itself. The actual generated PDF is for the full year leading up to the big day.

Conclusion

Frameworks

Remember Zappa in case you need a way to run Python on AWS Lambda.
- Also take a look at Chalice, which is another library for deploying Python to AWS Lambda. I haven’t tried it myself. It was written by Amazon themselves.
- Another notable framework is Serverless. Again, I haven’t tried this.
Consider using nanodjango for simple Django sites that might need to grow over time.

Creating a balanced investment portfolio with skfolio in Python

Tue, 08 Apr 2025 09:00:00 +0100

Nothing in this article is financial advice. I am not qualified to give financial advice. This is simply an educational article showing one way in which a portfolio could be balanced. No fitness for any purpose is expressed or implied.

Motivation

At the start of the tax year, it’s a natural time for those of us lucky enough to have savings to have a think about how they are invested. Recent turmoil in world equity markets makes this particularly relevant.

Like most retail investors, I think I’m better at choosing where to put my money than I actually am. The Dunning–Kruger effect is in full force, and I like to choose my own investments.

However, I do understand the need to spread risk by having a balanced portfolio. Being a Python developer, it is natural for me to wish to use Python to help me achieve this. This article highlights a reasonable approach that I am considering.

I shall present various code snippets throughout this post, but the full listing (repeating those snippets in context) is at the end.

Python libraries

We’ll be using several Python libraries, together with a simple Python script, to construct balanced investment portfolio.

I recommend using poetry for dependency management, in which case you can set things up in an empty directory by running:

poetry init (and accepting all the defaults)
poetry shell
poetry add skfolio
poetry add yfinance
poetry add matplotlib

Let’s have a quick look at the specific third-part libraries we’ll be using.

skfolio

skfolio, so called because it is built on top of scikit-learn / sklearn, will be doing most of the heavy lifting. It is a library specifically for portfolio optimisation, and has only been around since December 2023.

skfolio is essentially a toolkit for trying and comparing various portfolio theory models. It is a very powerful collection of tools, so we will only be touching on the basics of the functionality it offers in this article. Nevertheless, we will see how simple it can be to build a plausible investment portfolio that attempts to maximise returns and minimise risk.

yfinance

yfinance is a library for downloading financial data from Yahoo! Finance.

This is a very convenient way to get free historical data, as long as you need only daily prices.

Matplotlib

Matplotlib is the standard charting library familiar to most Python users.

We will use it to draw a custom chart of the matrix of correlations between the assets.

Declaring an asset universe

A significant task is defining the total set of assets – shares, bonds, exchange-traded funds (ETFs), etc. – we might consider investing in.

This will depend partly on what the broker makes available, and partly on personal preference (for example, you might choose not to consider equities in certain types of business, or sectors that you think are overvalued).

For the purpose of this article, I’ll choose a small selection of possible investments across equities, bonds and commodities. For using this code in practice I will expand this asset universe considerably: essentially, the more options the better. There are tools in skfolio for filtering down a large asset universe into a smaller one, for example by automatically discarding highly correlated assets, but that is outside the scope of this post.

Choosing the list of potential investments is a non-trivial task. We need to include the symbol used by Yahoo! Finance, which won’t necessarily match the one your broker provides. Generally some web searching is necessary, as well as making sure that the latest prices match up between your broker and Yahoo!

Here is a simple asset universe for us to get started with. Don’t draw any conclusions from this choice; I’m certainly not making any claims about how good or bad it would be to invest in any of these, but they do represent some variety. For practical purposes, you’d choose a much larger pool of potential investments to work with.

ASSETS = [
    # individual companies
    {
        "symbol": "TW.L",
        "name": "Taylor Wimpey",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "NG.L",
        "name": "National Grid",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "LLOY.L",
        "name": "Lloyds Bank",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "BARC.L",
        "name": "Barclays",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "MSFT",
        "name": "Microsoft",
        "groups": ["us", "equity", "tech"],
        "management_fee": 0.0,
    },
    # commodities
    {
        "symbol": "GBSS.L",
        "name": "Gold Bullion Securities",
        "groups": ["-", "commodity"],
        "management_fee": 0.4,
    },
    {
        "symbol": "BCOG.L",
        "name": "L&G All Commodities UCITS ETF",
        "groups": ["-", "commodity"],
        "management_fee": 0.02,
    },
    # bonds
    {
        "symbol": "0P0000KM23.L",
        "name": "Vanguard Global Bond Index Fund",
        "groups": ["world", "bonds"],
        "management_fee": 0.15,
    },
    {
        "symbol": "0P0000XBPM.L",
        "name": "Invesco Corporate Bond Fund (UK) Z (Acc)",
        "groups": ["uk", "bonds"],
        "management_fee": 0.5,
    },
]

Notes:

This dictionary format is just a convenient grouping of information for later manipulation.
The symbol has to match the symbol used by Yahoo! Finance. We use this to download the historic data for the asset.
The name doesn’t need to match Yahoo! Finance. It can be whatever makes sense to you.
The groups can be any labels that make sense to you. These will be useful for setting constraints (see the next section). Each group needs to have its entries in the same order (here we have used country, asset class and sub-class). It’s fine to omit entries (i.e. I’ve only marked up the tech sub-class); anything else will be automatically treated as None.
The management fee is given as an annual percentage. skfolio expects these to be fractions in the same granularity as the data (i.e. daily), so we’ll convert them later.

Setting constraints

One particularly handy feature of skfolio is how you can describe any hard constraints you want to impose on the portfolio as a set of readable strings.

LINEAR_CONSTRAINTS = [
    "tech <= 0.2",
    "uk >= 0.5",
    "us <= 0.1",
    "equity <= 0.7",
    "bonds >= 0.3",
    "commodity <= 0.3",
]

Here we’re imposing the following constraints:

tech investments (anything with “tech” in its groups) must constitute at most 20% of the portfolio.
at least 50% of the portfolio must be UK based
at most 10% of the portfolio may be US based
at most 70% of the portfolio may be invested in equities
at least 30% of the portfolio must be invested in bonds
at most 30% of the portfolio may be invested in commodities

You can also, if you wish, build up more complex conditions, such as uk >= us * 1.5 if you wanted at least one and a half times as much invested in the UK as in the US.

Note that we will also separately use this constraint to prevent more than 20% of the portfolio being invested in any one thing:

MAX_PROPORTION_IN_ONE_ASSET = 0.2

Optimising a portfolio with skfolio

There’s various glue code needed to fetch and transform the data, but let’s concentrate on the most interesting part… feeding the data into skfolio:

X = prices_to_returns(prices)                                         ①
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)  ②

model = MeanRisk(                                                     ③
    risk_free_rate=RISK_FREE_RATE / TRADING_DAYS_PER_YEAR,
    objective_function=ObjectiveFunction.MAXIMIZE_RATIO,              ④
    risk_measure=RiskMeasure.VARIANCE,                                ⑤
    min_weights=0.0,
    max_weights=MAX_PROPORTION_IN_ONE_ASSET,
    groups=groups,
    linear_constraints=LINEAR_CONSTRAINTS,
    management_fees=management_fees,
)
model.fit(X_train)

portfolio = model.predict(X_test)

It’s impressive how such a small amount of code is needed to achieve the bulk of what we need for creating a balanced portfolio of investments.

Let’s highlight some important parts:

① We convert from raw (closing) prices to linear daily returns: $\left( \frac{S_t}{S_{t-1}} - 1 \right)$.
② Split the historical data into ⅔ training and ⅓ test data. We’re not shuffling, so we’ll be training on old data and testing against more recent data.
③ Use the Mean-Risk Optimization estimator, which is a flexible, general-purpose optimiser.
④ Use the MAXIMIZE_RATIO objective function, which tries to maximise the Sharpe ratio of the portfolio.
⑤ We want the lowest variance portfolio that can achieve the high Sharpe ratio we find. Ideally we’d like a straight line going up, with no variance at all!

So, this set-up will try to maximize the Sharpe ratio subject to the constraints we set up, and find the portfolio make-up that minimises the variance to get there.

Output

As mentioned before, the full code is at the end of this article. But before we get to that, let’s take a look at the output it generates.

Historical price data

The first thing the script generates is a chart of the historic prices loaded from Yahoo! Finance.

Look out for any missing data or sudden jumps in price. (These happen more than you would expect because sometimes there’s a sudden switch in reporting between pounds and pence for UK assets. We set repair=True in the call to yf.download(), which tries to repair these automatically, but it doesn’t always get it right.)

Note that the Y-axis is logarithmic, which makes it easy to see all the asserts with their very different prices on the same chart.

Note also the clear effect of the 2008 market crash, and how some assets were impacted more than others. I wanted enough historic data so that the portfolio takes bad times into account as well as good.

Correlation matrix

The next window shown by the script is a (symmetric) correlation matrix, which shows for each asset how correlated it is with the others.

A high correlation (i.e. assets whose prices tend to move in the same direction) is shown in green or yellow. Every asset’s correlation with itself is 1.0, so the leading diagonal of the matrix is bright yellow.

An inverse correlation (i.e. assets whose prices tend to move in opposite directions) is shown in blue.

The third row, for Lloyds Bank, is a good example. Lloyds Bank shares are strongly inversely correlated with shares in the National Grid; but they are strongly positively correlated with shares in Barclays, which shouldn’t be too surprising.

Note also how the Vanguard Global Bond Index Fund and the L&G All Commodities UCITS ETF are very strongly inversely correlated.

So, at a very simple level, if all your investments were split across the two banking assets then you would have very low diversification and a high-risk portfolio (although one that would do very well when banking shares do well). Conversely, investing only across bonds and commodities is likely to give you very low risk – and losses in one area are likely to be offset by profits in the other – but at the same time it might be difficult to make a good return.

For my real portfolio I chose a much larger asset universe, and wanted to see a big spread of colours on this chart to give the algorithm plenty of scope for constructing a balanced portfolio.

Cumulative returns

The next chart shows what would have happened for the chosen portfolio in the test period.

It’s gone up, which is a good sign ;-)

Here, you’ll notice a tactical error. This chart is supposed to show the performance over the whole test period. But this was 30 years of history in a ⅔ training to ⅓ test split, so you’d expect 10 years of data to be shown.

If you look back at the historical price data, you’ll notice what’s happened. Not all the data sets go back to the start of the requested period. In particular, the Ivesco Corporate Bond Fund doesn’t start until 2019. So skfolio (specifically the call to prices_to_returns) is considering only the period with data for all assets, so effectively training on 4 years and testing on 2 instead of the expected training on 20 and testing on 10. I think that’s reasonable behaviour, but we need to be aware of it. In practice, the simplest workaround is probably to discard any assets that don’t stretch back far enough (see DISCARD_SHORT_DATASETS in the script below).

Portfolio composition

The final chart is the meat and potatoes, what we’ve been working towards. This shows which assets the algorithm has chosen, and in which proportions.

As noted before, this is definitely not investment advice! It’s not even how I plan to invest for myself, it being a cut-down asset universe. But it is indicative of the output you might get.

You can print out the portfolio’s make-up in python, or hover over the chart to see the tooltips.

In this run, the portfolio is comprised as follows:

20% Ivesco Corporate Bond Fund
20% National Grid
20% Gold Bullion Securities
10% L&G All Commodities
10% Vanguard Global Bond Index Fund
10% Microsoft
5% Barclays
3% Lloyd’s Bank
2% Taylor Wimpey

In this case, we had nine assets in our universe and skfolio has allocated some percentage of the portfolio to all of them. That won’t typically be the case for a more realistically large asset universe.

Let’s refer back to the constraints we set:

tech investments must constitute at most 20% of the portfolio.
- 10% Microsoft <= 20%
at least 50% of the portfolio must be UK based
- 20% Ivesco + 20% National Grid + 5% Barclays + 3% Lloyd’s + 2% Taylor Wimpey >= 50%
- Since it’s exactly 50% in this case, maybe this is a challenging condition to satisfy.
at most 10% of the portfolio may be US based
- 10% Microsoft <= 10%
- Again, since it’s exactly on the threshold, it’s likely that more would have been invested in Microsoft if not for this specific constraint.
at most 70% of the portfolio may be invested in equities
- 20% National Grid + 10% Microsoft + 5% Barclays + 3% Lloyd’s + 2% Taylor Wimpey <= 70%
at least 30% of the portfolio must be invested in bonds
- 20% Ivesco + 10% Vanguard >= 30%
- Another one on the borderline
at most 30% of the portfolio may be invested in commodities
- 30% Gold Bullion Securities + 10% L&G Commodities <= 30%
- And, again, right on the borderline.

So, it meets all the constraints that were set and seems like it would have done well if we had invested in such a portfolio for the last couple of years.

As the saying goes, past performance is no guarantee of future success (but it’s all we’ve got).

For those, like me, determined to pick their own investments, skfolio is certainly a more rigorous approach to take over stock picking. I can still control the universe of assets that I am willing to consider investing in, but have some guard rails in terms of imposing sensible constraints on the overall make-up of the portfolio, and some confidence that it will be balanced to maximise returns without undue risk.

Full script

Finally, as promised, here is the full python script I used in this article.

Remember, this is purely illustrative. I won’t even be using it myself in this form. I would want to add a much larger asset universe to choose from, flip the DISCARD_SHORT_DATASETS flag to True, and probably take a bit less history so that fewer assets get discarded, among other changes.

from datetime import date, timedelta, datetime
from pathlib import Path
from pprint import pprint
from skfolio import RiskMeasure
from skfolio.moments import EmpiricalCovariance
from skfolio.moments import EmpiricalMu
from skfolio.optimization import MeanRisk, ObjectiveFunction
from skfolio.portfolio import Portfolio
from skfolio.preprocessing import prices_to_returns
from sklearn.model_selection import train_test_split
from typing import Optional, Union
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import yfinance as yf

RISK_FREE_RATE = 0.02
TRADING_DAYS_PER_YEAR = 252
HISTORY_YEARS = 30  # want at least one recession in the training data

# Generally want this to be True, but for this sample script that will discard too many
# assets and the optimiser won't be able to find a solution.
DISCARD_SHORT_DATASETS = False

MAX_PROPORTION_IN_ONE_ASSET = 0.2
LINEAR_CONSTRAINTS = [
    "tech <= 0.2",
    "uk >= 0.5",
    "us <= 0.1",
    "equity <= 0.7",
    "bonds >= 0.3",
    "commodity <= 0.3",
]

# Management fees here are annual percentages, e.g. 1.0 represents a 1% annual fee.
ASSETS = [
    # individual companies
    {
        "symbol": "TW.L",
        "name": "Taylor Wimpey",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "NG.L",
        "name": "National Grid",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "LLOY.L",
        "name": "Lloyds Bank",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "BARC.L",
        "name": "Barclays",
        "groups": ["uk", "equity"],
        "management_fee": 0.0,
    },
    {
        "symbol": "MSFT",
        "name": "Microsoft",
        "groups": ["us", "equity", "tech"],
        "management_fee": 0.0,
    },
    # commodities
    {
        "symbol": "GBSS.L",
        "name": "Gold Bullion Securities",
        "groups": ["-", "commodity"],
        "management_fee": 0.4,
    },
    {
        "symbol": "BCOG.L",
        "name": "L&G All Commodities UCITS ETF",
        "groups": ["-", "commodity"],
        "management_fee": 0.02,
    },
    # bonds
    {
        "symbol": "0P0000KM23.L",
        "name": "Vanguard Global Bond Index Fund",
        "groups": ["world", "bonds"],
        "management_fee": 0.15,
    },
    {
        "symbol": "0P0000XBPM.L",
        "name": "Invesco Corporate Bond Fund (UK) Z (Acc)",
        "groups": ["uk", "bonds"],
        "management_fee": 0.5,
    },
]


def get_adjusted_close(
    ticker: str,
    start_date: Optional[date] = None,
    end_date: Optional[date] = None,
    cache_dir: Union[str, Path] = "cache",
) -> pd.Series:
    if end_date is None:
        end_date = date.today()
    if start_date is None:
        start_date = end_date - timedelta(days=HISTORY_YEARS * 365)

    cache_path = Path(cache_dir)
    cache_path.mkdir(parents=True, exist_ok=True)

    start_str = start_date.isoformat()
    end_str = end_date.isoformat()
    cache_file = cache_path / f"{ticker}_{start_str}_{end_str}.pkl"

    if (
        cache_file.exists()
        and datetime.fromtimestamp(cache_file.stat().st_mtime).date() == date.today()
    ):
        print(f"Reading {ticker} from cache.")
        data = pd.read_pickle(cache_file)
    else:
        print(f"Reading {ticker} from Yahoo Finance.")
        data = yf.download(
            ticker,
            start=start_str,
            end=end_str,
            repair=True,  # automatically cope with 100x switches between pounds and pence
        )
        if data is not None:
            data.to_pickle(cache_file)
        else:
            raise RunTimeError(f"Failed to download data for {ticker}. Stopping.")

    if "Adj Close" in data:
        return data["Adj Close"]
    else:
        return data["Close"]


def optimise(
    prices: pd.DataFrame,
    groups: dict[str, list[str]],
    management_fees: dict[str, float],
) -> None:
    # plot all the price data to check it's sensible
    prices.plot(logy=True)

    if DISCARD_SHORT_DATASETS:
        cols_to_drop = prices.columns[prices.iloc[0].isna()]
        print("Dropping assets where the data doesn't go back far enough:", list(cols_to_drop))
        prices = prices.drop(columns=cols_to_drop)

    X = prices_to_returns(prices)
    X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

    model = MeanRisk(
        risk_free_rate=RISK_FREE_RATE / TRADING_DAYS_PER_YEAR,
        objective_function=ObjectiveFunction.MAXIMIZE_RATIO,
        risk_measure=RiskMeasure.VARIANCE,
        min_weights=0.0,
        max_weights=MAX_PROPORTION_IN_ONE_ASSET,
        groups=groups,
        linear_constraints=LINEAR_CONSTRAINTS,
        management_fees=management_fees,
    )
    model.fit(X_train)

    portfolio = model.predict(X_test)
    pprint(portfolio.summary())

    returns = portfolio.plot_cumulative_returns()
    returns.show()

    plot_correlation_matrix(prices.corr(), list(prices))

    composition = portfolio.plot_composition()
    composition.show()


def plot_correlation_matrix(cov: np.ndarray, labels: list[str] = None) -> None:
    fig, ax = plt.subplots(figsize=(8, 6))
    cax = ax.matshow(cov, cmap="viridis")
    fig.colorbar(cax)

    if labels:
        ax.set_xticks(range(len(labels)))
        ax.set_yticks(range(len(labels)))
        ax.set_xticklabels(labels, rotation=90)
        ax.set_yticklabels(labels)

    ax.set_title("Correlation Matrix", pad=20)
    plt.tight_layout()
    plt.show()


if __name__ == "__main__":
    price_series = [
        get_adjusted_close(asset["symbol"]).rename(
            columns={asset["symbol"]: asset["name"]}
        )
        for asset in ASSETS
    ]
    prices = pd.concat(price_series, axis=1)

    for series in prices:
        print(f"Latest price for {series} is {prices[series][-1]}")

    groups = {asset["name"]: asset["groups"] for asset in ASSETS}

    management_fees = {
        asset["name"]: (asset["management_fee"] / 100) / TRADING_DAYS_PER_YEAR
        for asset in ASSETS
    }

    optimise(prices, groups, management_fees)

Edits:

2025-04-09 Removed pointless second constraint for the US from the full code listing.

Ghostly paragraphs in MkDocs

Sat, 12 Oct 2024 13:00:00 +0100

I really like MkDocs as a documentation tool. It’s based on Markdown, looks great (especially with the popular Material for MkDocs theme), has a great range of plug-ins, and can automatically rebuild and serve changes during local development.

Recently, I ran into a curious problem — here’s what happened, how I recreated it, and how you can fix it.

Ghostly goings-on

Background

Pages were mysteriously getting empty <p></p> HTML elements inserted, and I couldn’t figure out why.

The set-up

Let’s go step by step through how to reproduce the problem I was seeing by building an empty MkDocs project from scratch in a Docker container.

docker run -it --rm -p 8000:8000 python:latest bash

# pip install mkdocs

...

cd home
mkdocs new spooky
cd spooky
mkdocs serve --dev-addr 0.0.0.0:8000 > /dev/null 2>&1 &

Next, browse to http://127.0.0.1:8000/ on the host machine. You should see a default page being served from inside the Docker container.

Reproducing the problem

Now, let’s replace the default content with a setup that demonstrates the issue with “ghostly” empty elements. We want a paragraph of text, followed by an HTML table where the content should be treated as Markdown. Although slightly contrived, this example is based on a real problem seen in the wild.

While Markdown’s own table syntax is simpler for basic tables, using HTML tables offers greater flexibility, especially for complex layouts. So that is what we will be using, with the md_in_html extension to allow markdown content inside the tables. Let’s enable it.

Press Ctrl+C to kill the mkdocs process, then update the MkDocs settings like this:

echo -e "\nmarkdown_extensions:\n  - md_in_html" >> mkdocs.yml

Next, we want to install an editor to make it easy to modify the Markdown source file.

apt update

apt install vim

vim docs/index.html

Change the content as follows. Reload the web page and note how the corresponding HTML has been generated.

# Example

Paragraph before.

<div id="outer" markdown="1"><div id="inner" markdown="1">Some **bold** text.</div></div>

Paragraph after.

<h1 id="example">Example</h1>
<p>Paragraph before.</p>
<div id="outer">
<p><div id="inner" markdown="1">Some <strong>bold</strong> text.</p>  ⮄ paragraph AROUND div 😈
</div>
</div>
<p>Paragraph after.</p></div>

Note the unexpected <p> and </p> tags.

However, notice what happens if we add some line breaks. The same Markdown content now produces different HTML output:

# Example

Paragraph before.

<div id="outer" markdown="1">
<div id="inner" markdown="1">Some **bold** text.</div>
</div>

Paragraph after.

<h1 id="example">Example</h1>
<p>Paragraph before.</p>
<div id="outer">
<div id="inner">
<p>Some <strong>bold</strong> text.</p>  ⮄ paragraph INSIDE div 😇
</div>
</div>
<p>Paragraph after.</p></div>

Now the content of the inner div contains a paragraph of text, but there are no <p></p> tags wrapping the div, which is reasonable behaviour.

So, to clarify, the original looked like <p><div>...</div></p> and, just by modifying the line breaks in the Markdown, the HTML output changed to <div><p>...</p></div>.

You can take it a step further by specifying the inner div as a span element (i.e. inline rather than block-level) like this:

# Example

Paragraph before.

<div id="outer" markdown="1">
<div id="inner" markdown="span">  ⮄ note the span
Some **bold** text.
</div>
</div>

Paragraph after.

<h1 id="example">Example</h1>
<p>Paragraph before.</p>
<div id="outer">
<div id="inner">
Some <strong>bold</strong> text. ⮄ no paragraph tags at all 😇
</div>
</div>
<p>Paragraph after.</p></div>

Does it matter?

What prompted this post was a “bug” where empty <p></p> appeared before a table with markdown="1". These ghost paragraphs seemed to come out of nowhere. Even though my MkDocs theme was suppressing display of these empty paragraphs, they were still affecting the CSS styling when printing because my heading had “keep with next” styling: these empty paragraphs became the next elements, so headings could still appear at the bottom of printed pages.

The examples in this post are a simplified version (I had difficulty reproducing the original issue fully). This issue took a lot of debugging and head-scratching to resolve, and I found little help online. Hopefully, this post will save someone else from the same frustration.

So, what’s going on?

The CommonMark spec attempts to unambiguously define the syntax of Markdown. The section about HTML blocks goes into a lot of detail about HTML in Markdown, including many examples.

There are several layers of things happening here:

Processing of HTML blocks in Markdown content is part of standard Markdown processing. At this level, any Markdown syntax inside the HTML content will be treated literally as HTML content.
- The HTML content will just be identified and then passed through to the rendered page unchanged. It is not parsed, and need not be valid.
- Blank lines will be treated as the end of an HTML block, and cause the Markdown processor to switch back to rendering Markdown (with unexpected consequences if you have blank lines inside the HTML block).
The second layer involves processing the Markdown content inside HTML blocks inside Markdown. It is triggered by the markdown="..." attribute, which is supported by some markdown processors. In the case of MkDocs, this is provided by python-markdown’s md_in_html extension (which we explicitly enabled in the MkDocs settings above).
- Note that using this extension will change the HTML tag processing slightly. Since the extension needs to parse the content anyway to differentiate between HTML and Markdown content, it treats the HTML less literally and will do things like automatically closing dangling HTML tags.

So, the spooky paragraphs are appearing for elements that are being erroneously interpreted as block-level elements because of the intricacies of the Markdown parsing. But, as demonstrated by these examples, you can control this with careful formatting.

Summary

When working with HTML blocks in MkDocs Markdown, follow these guidelines to avoid unexpected issues:

Always include a blank line before and after each HTML block.
Avoid blank lines inside the HTML blocks.
Start each block-level HTML element (start and end tag) on a new line.
Avoid indenting any of the lines of the HTML block at all.

Additionally, when you want to render content as markdown inside those HTML blocks:

Mark the HTML element that wraps the markdown content with markdown="1". Work outwards from there to each parent HTML element, adding the markdown attribute there each time until you reach the outermost HTML element.
If you get unexpected block-level elements being created, this is probably because something (probably the HTML element immediately wrapping the markdown content) is being interpreted as a block-level element. To fix this, change its markdown="1" attribute to markdown="span" to ensure it’s treated as inline content.

Python logging: basic, better and best

Tue, 04 Jun 2024 13:00:00 +0100

In this post I’ll show several approaches to logging in Python.

We’ll start off with the simplest, most basic approach: using print statements. This is often looked down on, but it does have some real utility and there are a few useful hints you can use to get the most of this approach.
Next, we will move onto a more formal approach of using Python’s built-in logging capabilities.
Finally, we’ll look at the structlog library, and talk about why I consider it to be the best approach. We’ll also look at the pros and cons of some other third-party logging libraries, and why I would choose structlog over them.

Basic: print statements

Sometimes, a print statement is all you need to quickly verify your code’s behaviour.

In a pinch, you might only want confirmation that the code you are running is entering a particular code block, such as the expected branch of an if statement. For instance, I often use print("sanity") or print("got here").

To trace the path through some code, scattering print(“A”), print(“B”), etc., can help. This is especially effective if an exception is raised and you’re having difficulty tracking down the cause.

This method is quick and dirty, with quickness being the key. It’s not elegant but can be effective and typically won’t hit source control.

As an aside, using the f-string syntax with a variable name followed by = is a useful shortcut. Instead of writing print("answer=", answer), you can simply do this:

>>> answer = 42
>>> print(f"{answer=}")
answer=42

I use prints for transient testing and debugging. For more permanent solutions, logging is a better choice.

Better: logging

Base logger

Python has a logging component in the standard library. This is straightforward to use and configure, and well documented.

It is the benchmark against which all third-party logging libraries must be judged; each of them addresses aspects of the standard logging library that their creators think can be improved.

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)
logging.info("Hello, world.")

2024-05-18 22:13:06,880 - INFO - Hello, world.

There are various logging levels you can use for messages, in increasing order of severity:

logger.debug('This is a debug message')
logger.info('This is an info message')
logger.warning('This is a warning message')
logger.error('This is an error message')
logger.critical('This is a critical message')

You can adjust the logging level to control which messages are logged. For instance, you might set a lower level when debugging to see more detailed messages, and a higher logging level in production to log only warnings and above.

A note about message construction

When constructing log messages, there are different ways to include variable data. Consider the following examples:

widget = 3
logger.info(f"Widget {widget} is out of space")
logger.info("Widget %s is out of space", widget)

Both these logger calls output Widget 3 is out of space. However, the second format is preferable for a couple of reasons:

The conversion of widget to a string and embedding it into the larger message string will only happen if the log level is low enough for the message to be output. This can save processing time and resources when logging is disabled for that level.
Some formatters will be able to group together all the messages using the same template. This is important if you use a log aggregation service that you will later want to search through and generate statistics from.

Custom logger

In full-fledged applications, it’s beneficial to have each Python module maintain its own independent logger. This approach offers several benefits:

You can set log levels and handler types independently in different parts of your code.
Log messages can include the logger name to highlight which part of your codebase they came from.
Your logging won’t interfere with the other code that calls into yours (for example, if you are creating a library).

Here is a typical example. It creates a logger with two handlers: one to log to the console all messages at level INFO and above, and one to log to a file all messages at level DEBUG and above.

Note that the base logger itself is set to DEBUG; if it were set to a higher level then debug messages would be filtered out before getting to the handlers, whatever their own logging levels were set to.

import logging

logger = logging.getLogger(__file__)
logger.setLevel(logging.DEBUG)

console_handler = logging.StreamHandler()
file_handler = logging.FileHandler("app.log")

console_handler.setLevel(logging.INFO)
file_handler.setLevel(logging.DEBUG)

formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
console_handler.setFormatter(formatter)
file_handler.setFormatter(formatter)

logger.addHandler(console_handler)
logger.addHandler(file_handler)

logger.debug("Debug message to file")
logger.info("Info message to console and file")

matthew@grug ~/p/2024-05-15> python3 main_logging.py
2024-05-18 22:56:16,249 - /home/matthew/permatmp/2024-05-15/main_logging.py - INFO - Info message to console and file
matthew@grug ~/p/2024-05-15> cat app.log
2024-05-18 22:56:16,249 - /home/matthew/permatmp/2024-05-15/main_logging.py - DEBUG - Debug message to file
2024-05-18 22:56:16,249 - /home/matthew/permatmp/2024-05-15/main_logging.py - INFO - Info message to console and file

Typically, you’d write a utility function to apply your preferred setup to each logger.

def setupLogging(log_name):
    logger = logging.getLogger(log_name)
    logger.setLevel(logging.DEBUG)

    console_handler = logging.StreamHandler()
    file_handler = logging.FileHandler("app.log")

    console_handler.setLevel(logging.INFO)
    file_handler.setLevel(logging.DEBUG)

    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    console_handler.setFormatter(formatter)
    file_handler.setFormatter(formatter)

    logger.addHandler(console_handler)
    logger.addHandler(file_handler)

    return logger

Then you would set up the logs in your modules by calling it:

from utils import setupLogging

logger = setupLogging(__file__)

Best: structlog

structlog is a powerful and flexible logging library that goes above and beyond the functionality of Python’s built-in logging. It focuses on structured logging, where simple log messages (events) accompanied by value-key metadata (context), making them easy for other tools to parse later.

Structured logging separates the context from the message, making the logs easier to write and parse. While you can still use %s placeholders (as described above), structlog’s approach simplifies adding context to your logs.

>>> from structlog import get_logger
>>> logger = get_logger()
>>> widget = 3
>>> logger.info("Widget is out of space", widget=widget)


2024-06-02 21:46:31 [info     ] Widget is out of space         widget=3

Binding values

One of the powerful features of structlog is the ability to bind values to loggers. This means you can attach extra context, such as a session ID, to a logger, and these values will automatically be included in all subsequent log messages.

>>> from structlog import get_logger
>>> logger = get_logger()
>>> bound_logger = logger.bind(session=123)
>>> bound_logger.info("hello")


2024-06-04 11:37:49 [info     ] Hello                          session=123

Exceptions

When debugging locally, the more quickly you can get to the source of the problem the better. Structlog’s display of stack traces is clear and helpful, making it easier to diagnose issues.

Here’s an example of a stack trace generated by structlog:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/matthew/permatmp/2024-05-15/main_exception.py:4 in <module>                                │
│                                                                                                  │
│   1 from structlog import get_logger                                                             │
│   2 logger = get_logger()                                                                        │
│   3 try:                                                                                         │
│ ❱ 4 │   div_by_zero = 1 / 0                                                                      │
│   5 except BaseException as e:                                                                   │
│   6 │   logger.exception(e)                                                                      │
│   7                                                                                              │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │          e = ZeroDivisionError('division by zero')                                           │ │
│ │ get_logger = <function get_logger at 0x75c44db8dd80>                                         │ │
│ │     logger = <BoundLoggerLazyProxy(logger=None, wrapper_class=None, processors=None,         │ │
│ │              context_class=None, initial_values={}, logger_factory_args=())>                 │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ZeroDivisionError: division by zero

As you can see, structlog highlights the problematic line with a red arrow, along with some context around it. It also lists all the local variables (locals), which can be useful for diagnosing the issue.

Aside: this colourful HTML was generated by piping into ansi2html with --partial --inline parameters set.

Typical structlog config

Structlog’s configuration is incredibly flexible. The documentation is thorough but fragmented, so here I’ve consolidated guidance into a realistic configuration that includes the following features:

Optimised for human readability when running locally; optimised for aggregation and machine readability when running in production.
Adds the file name and line number to log messages, which isn’t included by default but is generally useful information.
Uses a LOG_LEVEL environment variable to set the logging level; log entries at lower levels will be treated as no-ops.
Uses the orjson library for JSON generation in production, for ultra-fast JSON creation (straight to bytes).

import logging
import os
import sys

import orjson
import structlog
from structlog.contextvars import merge_contextvars
from structlog.dev import ConsoleRenderer, set_exc_info
from structlog.processors import (
    CallsiteParameter,
    CallsiteParameterAdder,
    EventRenamer,
    JSONRenderer,
    StackInfoRenderer,
    TimeStamper,
    add_log_level,
    dict_tracebacks,
)

shared_processors = [
    # This list is based on _BUILTIN_DEFAULT_PROCESSORS.
    # See https://github.com/hynek/structlog/blob/main/src/structlog/_config.py
    merge_contextvars,
    add_log_level,
    StackInfoRenderer(),
    set_exc_info,
    # Extras not in the defaults
    CallsiteParameterAdder([CallsiteParameter.MODULE, CallsiteParameter.LINENO]),
]

# Approach based on https://www.structlog.org/en/stable/logging-best-practices.html
logger_factory = None  # use the default
if sys.stderr.isatty():
    # Pretty printing when we run in a terminal session.
    # Automatically prints pretty tracebacks when "rich" is installed
    processors = shared_processors + [
        TimeStamper(fmt="%Y-%m-%d %H:%M:%S", utc=False),
        ConsoleRenderer(),
    ]
else:
    # Print JSON when we run, e.g., in a Docker container.
    # Also print structured tracebacks.
    processors = shared_processors + [
        # slightly less readable and more easily processed, e.g. `2024-06-02T23:04:00.000000Z`.
        TimeStamper(fmt="iso", utc=True),
        EventRenamer("msg"),  # rename "event" key to "msg"
        dict_tracebacks,
        JSONRenderer(serializer=orjson.dumps),
    ]
    # Note that orjson generates bytes not strings
    # logger_factory=structlog.BytesLoggerFactory()  # for outputting JSON to console
    logger_factory=structlog.BytesLoggerFactory(
        file=open("output.log", "ab")  # for outputting JSON to file
    )

log_level_raw = os.environ.get("LOG_LEVEL", "INFO").upper()
log_level = logging.getLevelName(log_level_raw)
if isinstance(log_level, str):
    raise ValueError(f"Invalid log level: {log_level_raw}")


structlog.configure(
    cache_logger_on_first_use=True,
    wrapper_class=structlog.make_filtering_bound_logger(log_level),
    processors=processors,
    logger_factory=logger_factory,
)

The configuration above should be run once in your application. Then, in the modules where you use logging, you can set up and use the logger as follows:

import structlog

logger = structlog.get_logger()
widgets = 3

logger.debug("Too low a level -- will be skipped");
logger.info("Test info", widgets=widgets)
logger.warning("Test warning", widgets=widgets)
logger.error("Test error", widgets=widgets)
logger.critical("Test critical", widgets=widgets)

2024-06-02 23:04:00 [info     ] Test info                      lineno=5 module=my_module widgets=3
2024-06-02 23:04:00 [warning  ] Test info                      lineno=5 module=my_module widgets=3
2024-06-02 23:04:00 [error    ] Test info                      lineno=5 module=my_module widgets=3
2024-06-02 23:04:00 [critical ] Test info                      lineno=5 module=my_module widgets=3

Or, if outputting to a file:

> cat output.log
{"widgets":3,"level":"info","module":"main_structlog2","lineno":75,"timestamp":"2024-06-04T09:20:57.636807Z","msg":"Test info"}
{"widgets":3,"level":"warning","module":"main_structlog2","lineno":76,"timestamp":"2024-06-04T09:20:57.637064Z","msg":"Test warning"}
{"widgets":3,"level":"error","module":"main_structlog2","lineno":77,"timestamp":"2024-06-04T09:20:57.637170Z","msg":"Test error"}
{"widgets":3,"level":"critical","module":"main_structlog2","lineno":78,"timestamp":"2024-06-04T09:20:57.637254Z","msg":"Test critical"}

A note about async logging

Structlog also supports async logging, which can be useful in scenarios where logging is particularly slow and frequent, such as logging to a database. The async function calls use a prefixes, like await logger.ainfo("...").

Async logging can be beneficial when logging involves complex pipelines or expensive-to-write syncs. However, it’s important to combine this with batching, saving up log entries and writing them in bulk periodically. Otherwise, the overhead of async call machinery might outweigh the benefits of asynchronous writing, potentially resulting in worse performance.

As with most performance tweaks, you should profile the code to check that it works in practice before adopting this wholesale.

Other options

Loguru

Loguru is a good alternative logging framework that simplifies the logging setup in Python.

Pros:

Much easier configuration that structlog.
Faster than the standard Python logging module.

Cons:

Uses string-based logging with {} placeholders; I would prefer full structured logging, and this form is not drop-in compatible with existing standard logging implementations using %s style placeholders.

Logbook

Logbook is another interesting Python logging library that offers unique features and improvements over the standard logging module.

Pros:

Faster than the standard Python logging module.
Includes esoteric handlers for logging directly to some unusual syncs. For example, you can log to Twitter or ZeroMQ, or send critical errors to an mail eaddress.
Trivially easy to redirect standard logging calls to Logbook with a redirect_logging() call.

Cons:

Does not support full structured logging.

Sentry

There is a Python API for the Sentry platform, which might be of interest if you want to aggregate all your logs into Sentry.

python-json-logger

The python-json-logger library adds JSON-formatted output to the standard logging module. However, it doesn’t appear to be under active maintenance.

Wrap-up

Structlog offers some welcome extra functionality over that provided by the standard library’s logging component. The coloured output makes it easier to parse console output; in particular, the formatting of logged exceptions makes it simpler to track down problems.

Logging is one of those things that you set up once in a project and then rarely have to tweak the configuration again. While structlog’s flexibility can make its documentation challenging to follow, this initial setup effort pays off with simplified maintenance in the long run.

Although many other logging libraries are easier to configure, they primarily focus on filling in templated message strings rather than providing JSON-formatted output. As demonstrated in this post, structlog makes it possible to achieve both easy-to-read logging locally and machine-friendly output in production.

Cloudflare DNS and Site44

Tue, 07 May 2024 13:00:00 +0100

The problem: could not load this blog without the www. prefix

This blog is hosted on site44. This is a very convenient way of running a static website – simply update your static content in Dropbox and Site44 will serve it up as a site for you. It’s reasonably priced and I’ve been using it for various things for years.

One long-standing quirk has been that you have to specify the URL when setting up the site. I chose www.matthewstrawbridge.com, which meant that while you could browse to http://www.matthewstrawbridge.com or https://www.matthewstrawbridge.com and everything would work, omitting the www. prefix would just load Site44’s 404 error page:

Note that part of the complication is that Site44 requires a CNAME record in your DNS to do the redirection, rather than simply having an A record pointing at your site.

See this snippet from their custom domain help page:

CNAME aliases do not apply to “naked” domains like example.com—they only apply to subdomains like www.example.com or blog.example.com. Site44 only supports subdomains; use “forwarding” (see below) to handle naked domains.

This is a record of how I solved this problem.

The solution: adding a CloudFlare redirect rule

CloudFlare is renowned for providing the infrastructure and know-how to keep large sites up and available online, with a good reputation for preventing denial-of-service (DOS) attacks. But alongside this commercial clout, they also offer a free plan which includes a DNS service.

There are three things necessary to get everything working.

1. A `CNAME` link from the custom domain to Site44

This was already present on my previous DNS host, and CloudFlare imported it and set it up automatically. This is a CNAME entry from www to site44. Once this is in place (and propagated) you can navigate to the site when using the www. prefix.

Note that this has the proxying option OFF, which seems to be necessary for Site44.

The CNAME entry is shown on the bottom line in this screenshot:

2. An `A` record entry for the top-level host

In theory this shouldn’t be necessary, but CloudFlare’s redirection rules for the top-level site (see next step) don’t seem to work without it.

This is the other record shown in the screenshot above; an A DNS record to the top-level site (you can just enter @ into the relevant box, and Cloudflare will fill in the host – in this case matthewstrawbridge.com – automatically.

The content that it’s redirecting to shouldn’t really matter, because the redirection rule we create will kick in before anything ends up at that IP address. I chose to use 192.0.2.1, which is a dummy address used in lots of examples.

Note that this record can have CloudFlare’s automatic proxying turned on (shown by the orange icon).

3. Create the redirection rule

Now we want any attempts to access the top-level host to be automatically redirected to the equivalent URLs with the www. prefix.

This was a bit more complex.

From the Cloudflare menu, select Rules > Redirect Rules. This is preferable to using page rules, which I understand to be deprecated.

Then you can create a redirection rule that looks like this:

It uses a custom filter expression as follows:

when the hostname in the incoming request is matthewstrawbridge.com (rather than, say, www.matthewstrawbridge.com) then
respond with a permanent redirect (301 HTTP status code) to the alternative URL that you get by concatenating www.matthewstrawbridge.com with the rest of the path from the original URL (e.g. /some-article/) and preserving the query string (e.g. ?search=wibble) if there is one.

Conclusion

All this is a little bit more long-winded than I’d like, but at least it’s a one-off cost.

As always with DNS things, it can take a long time for the settings to propagate.

I guess the popular alternative these days for a technical blog such as this would be to host it on GitHub pages, but I’ve been happy with Site44 and this fixes the main niggle that I’ve had with it.

Thoughts on writing code well

Sun, 24 Jan 2016 12:00:00 +0000

About this post

I had intended to write a post about software craftsmanship in general. Instead, the topic of test-driven development/design (TDD) seems to have taken over.

The irony isn’t lost on me. You don’t always know at the outset of creating something what form the final product will take. TDD is a good fit for this: you’re not guessing what you might need, just creating the next small thing you know you need in the simplest way possible. The final product can look after itself.

Test-driven design (TDD)

Degrees of hatred and fandom

How do you feel about unit testing? It seems to be a topic that divides developers.

I’ve always been a fan of having at least some unit tests, but over the last few years I’ve moved from “use common sense and write just enough good tests afterwards” to “use strict test-driven design”.

Everyone fits somewhere on the scale between “writing any tests at all is a complete waste of time” to “all code should be written using TDD”. The purpose of this article is to encourage you to nudge to the right on this scale; at least try the next step and see how good a fit it is for the projects you work on.

Definitions

Uncle Bob Martin defines the Three Laws of TDD as follows:

You are not allowed to write any production code unless it is to make a failing unit test pass.
You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.
You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

His Clean Coders videos give a good and entertaining introduction to TDD and other important topics in this area. The key point is that you are constantly in a tight loop: red (failing test), green (passing tests), refactor.

The three rules above are simple but they are not simplistic. It takes practice to write code in this way. If you’ve been writing code for years, following these rules seems unintuitive, even patronising. It’s almost as if they were designed to slow you down.

But you get all sorts of benefits if you apply this strictly. Ten years ago (so this isn’t exactly a new idea), Tim King wrote an article listing a dozen of these¹.

If you only do manual testing, or no testing at all, then you are doomed to a lifetime of chasing bugs in production code. You’re climbing a mountain without a safety rope.

Seriously, if you think TDD is a waste of time and effort, or something that’s only useful for newbie programmers, I’d encourage you to try it out on your next pet project. (If you don’t have a suitable project coming up, try Roy Osherove’s String Calculator TDD kata.) I think you’ll be pleasantly surprised by the positive effect it has on the code under test and how many times you’ll find you’ve uncovered a bug during development that would otherwise have made it into the wild. However, it does require a degree of humility to get started; a willingness to accept that, just maybe, there’s a better approach than the one you’ve been using for years.

Killing disease at the point of contraction

How unit tests stop bugs

It’s all very well saying that TDD helps guide the design of the product, but you probably care more about what it can do to help prevent bugs. After all, the tests are there to check behaviour and uncover bugs.

TDD has a dual role when it comes to finding bugs. Firstly, as part of the process, you’re continually writing new tests for behaviour that doesn’t yet exist, then modifying the code until the test passes. Sometimes you’ll run the new test, expecting it to pass, but it fails. This is a genuine bug; because you’ve caught it at the point of injection, it’s probably easy to fix. Wipe your brow and move on, but remember to be thankful that the test was there. Just think how long it might have taken to track the problem down from a customer’s patchy bug report six months after deploying the code!

The second role TDD plays is in the construction and frequent running of a comprehensive suite of regression tests. If you break something, you should find out straight away. That’s very powerful.

Avoiding bugs altogether where you can

Testing isn’t a silver bullet. All useful code has a non-zero bug density.

But code that doesn’t exist does have zero bugs. You just can’t get better than that! It’s why principles such as DRY (don’t repeat yourself) and YAGNI (you aren’t gonna need it) are important: having code once is better than having it twice; having dead code rotting alongside living code is harmful and confusing.

Poor developers write buggy code. Better developers write fewer bugs. Great developers avoid writing the code at all.

If there are bugs, finding them early

Use automated methods to catch simple problems first. Then use code review to catch even more. There are some fine static analysis tools that you can run over your code to find problems early, and to avoid code reviewers from being bogged down with finding issues that could have been caught by a tool:

.NET
- Resharper
- FxCop
- StyleCop
- NDepend
JavaScript
- JSLint

Finding bugs early is also why I prefer strong typing over dynamic typing. I’d rather find bugs at compile-time than at run-time, and I’ve never wanted to redefine a string as an int in mid flow.

Simplicity

It all comes back to simplicity.

Source code has a natural tendency to become more complicated over time. Hacks and other special cases will be added. Documentation will get out of sync with the code. A dozen people will edit the same file, applying subtly different coding styles to it each time. Classes will grow ever bigger and take on more and more responsibilities. Some code will become uncallable or uncalled and start to rot.

It takes energy to fight against this complication.

TDD can help. Constant, safe refactoring is a force for keeping the code simple. Making just one focused change at a time is working in a simple way.

Static code analysis can also help. It helps to enforce good practice across the code base, which makes its maintenance more straightforward. Some level of automated analysis eases the burden on code reviewers, so they can concentrate on the more complex issues that can’t be found automatically.

I believe that these three quality gates – the use of TDD to guide design and create a regression framework, static analysis to find some problems automatically and code review to find still more – all help to minimise the number of bugs escaping into the wild. Which leaves developers with more time to do useful work and less time chasing their tails fixing bugs in production code.

Footnotes

* Unit tests prove that your code actually works * You get a low-level regression-test suite * You can improve the design without breaking it * It’s more fun to code with them than without * They demonstrate concrete progress * Unit tests are a form of sample code * Test-first forces you to plan before you code * Test-first reduces the cost of bugs * It’s even better than code inspections * It virtually eliminates coder’s block * Unit tests make better designs * It’s faster than writing code without tests! ↩

The first software I installed on my new Windows laptop

Sun, 19 Jul 2015 13:00:00 +0100

After many years service my Windows laptop gave up the ghost. It was literally starting to fall apart (the plastic near the hinge was breaking and it would occasionally freeze up completely). So, as I had to reinstall the software I use on my new laptop, I thought it would be interesting to record what I installed and in which order.

This is a useful aide memoire for me, and might flag up a few interesting tools that you might not know about.

1. Chrome

There are only two good uses for Internet Explorer: downloading another browser and IE compatibility testing.

So the first thing I did after powering on my new laptop was to open IE and use it to install Google Chrome.

As well as the browser itself, I created virtual application shortcuts – | More tools | Create application shortcuts… – for some of the sites I access most often:

GMail
eBay
Facebook
Twitter

I also set Chrome to start with a blank page and to use DuckDuckGo as my default search engine.

2. Paint.NET

The next application I installed was Paint.NET. It’s always useful to have a free graphics manipulation package, and I find Paint.NET a lot simpler to work with than GIMP.

3. Dropbox

I love Dropbox. It just sits there quietly ticking away, making sure all my important files are backed up.

Just by installing Dropbox and letting it pull down my files, I ended up much more at home in my new laptop’s environment.

4. Vim

Next came Vim, the text editor I couldn’t be without.

I copied my _vimrc file out of Dropbox and immediately had a text editor set up just the way I like it.

5. ConEmu

ConEmu is a console emulator. It’s got lots of bells and whistles, but the main benefits for me are that you can freely resize the window, and you can group multiple consoles together as tabs within a single application. Having a split window is also useful on occasion.

6. Inconsolata font

My favourite fixed-width font is Inconsolata – in particular, the tweaked version with straight quotes. This is the font I always use in Vim on my own machines.

7. LinqPad

LinqPad is a strange little application. It describes itself as The .NET Programmers Playground. I find it’s really handy when I just want to try something out in C#. It’s quicker to open LinqPad than to create a temporary solution in Visual Studio, and the rich formatting of the .Dump() function that it adds to all objects makes it easy to see what’s going on. It’s particularly useful when messing about with regular expressions.

8. Beyond Compare

Not free, but cheap enough and good enough that it’s a no-brainer: Beyond Compare does one thing and does it well – comparing files.

Specifically, it has some novel features:

Explorer integration, so you can right-click a file and specify it as the left file to compare, then navigate somewhere else entirely, right-click that file and compare with left
Built-in support for comparing images and spreadsheets.
A text compare view where you can paste in snippets of text from wherever you want to grab them from. This is insanely useful and I find myself using it all the time (e.g. for resolving merge conflicts in source control).

9. Directory Opus

Just as ConEmu is a command window replacement, Directory Opus is a Windows Explorer replacement.

It does all the stuff you’d expect, and it does it well.

10. Scrivener

Scrivener is the best application for working with fiction.

11. Process Explorer

Once you’ve tried Process Explorer the standard Windows Task Manager just won’t cut it any more.

Originally developed by SysInternals, it’s now been subsumed into Microsoft.

12. Visual Studio

Speaking of Microsoft, I also installed Visual Studio…

13. Office

… and Microsoft Office.

14. Adblock plus

There’s only so long you can use Chrome before you remember you haven’t got Adblock Plus installed. I remembered, so I installed it.

Seriously, bad marketeers are their own worst enemies! I left the “allow unobtrusive ads” option enabled – I wish the whole fiasco wasn’t necessary.

15. FoxIt Reader

If you need to read PDFs but don’t need the bloat and incessant updates that you get with Adobe Reader then FoxIt Reader is a lightweight alternative.

16. VLC Player

Similarly, VLC is a media player that can handle almost any file format while being fairly unobtrusive.

17. F.lux

F.lux is something I’ve only discovered fairly recently. It puts an overlay on your display to change the colour temperature to better suit artificial lighting in the evening. It’s an interesting idea.

18. 7-Zip

For archiving I use 7-Zip. Not only does it produce really tiny files in its own .7z format, but it can also produce smaller standard .zip files than most other programs.

19. Cygwin

There are a few Unix commands I don’t want to be without on Windows, so I installed CygWin and its apt-cyg package manager.

Here are some worthwhile commands that aren’t installed by default but are worth getting:

wget
whois
par

20. GitHub for Windows

It used to be really painful to work with GitHub from a Windows PC. You had to install all sorts of third-party software and manage your own private key.

These days, the GitHub for Windows handles all that fluff and makes it easy to stay in sync.

It actually includes both MySys Git and PoshGit, which will be in the path as long as you open a shell from the application, so there’s nothing else to install.

Except…

21. TortoiseGit

Okay, sometimes it’s useful to be able to interact with Git from an Explorer (or Directory Opus) window instead of from the command line. TortoiseGit isn’t quite as polished as TortoiseSVN, but it does a decent job.

Rather than installing another MySys Git, you can just type which git from a command window opened up via GitHub for Windows and specify that path. If you check the path, Tortoise will complain because git --version doesn’t contain “mysys”, but that’s just because GitHub have rebranded it slightly. It’ll still work fine.

I ended up with this:

C:\Users\Matthew\AppData\Local\GitHub\PortableGit_c2ba306e536fdf878271f7fe636a147ff37326ad\bin

22. Resharper

Last, but not least, I find it increasingly frustrating to try to use a Visual Studio environment without Resharper installed (for example, when pairing with someone else). If you use Visual Studio you owe it to yourself to use Resharper.

Yes, it slows things down slightly. No, I don’t care, because it gives back much, much more time in improved productivity.

Conclusion

So there we have it. A smorgasbord of quality software – some mainstays, and a few you might not have heard of. Many of them are free or inexpensive. All of them improve my day-to-day productivity.

I hope you found something new that you’d like to investigate. Is there something missing that you couldn’t imaging being without on your own PC? If so, drop me a line on Twitter and let me know.

Migrating from Drupal to Jekyll

Sat, 08 Nov 2014 23:30:00 +0000

What’s this blog post about?

The content of this blog has been migrated from one platform to another several times now. Back in the day I had an account on Blogger. It looks like it’s still there, which probably isn’t doing wonders for the SEO on this site with the copied content.

After that I started making blog posts on Software Reference, my publishing company’s site. The posts were technical things that interested me, but weren’t necessarily relevant to the topics of the books we published (i.e. they tended to be about programming rather than netiquette or Microsoft Word 2007).

Next the content moved to its own domain: matthewstrawbridge.com. Until now it’s been a Drupal site. This post is about migrating that content from a dynamic Drupal site to a static one generated using Jekyll… why I wanted to change and how I did it.

What’s wrong with Drupal?

First things first, what is Drupal

Drupal is a content management system along the same lines as Joomla and WordPress.

I started using Drupal because it seemed to be the most powerful of these options: it gives you low-level control of your site without tying you into a particular way of doing things, and there is a good selection of plug-ins for further extending it.

There’s nothing inherently wrong with Drupal. It does what it sets out to do. Once you’ve got a site set up you can post new content and it takes care of the rest. If you’ve got some hosting space you can get it set up and configured fairly easily.

What’s the catch?

However, the elephant in the room is the way it handles updates.

How I would like Drupal to handle updates:

You log in as the administrator and click a button labelled Update.

How you actually handle updates:

Take the site offline.
Manually download the latest version.
Remind yourself how to install it.
Go through a painful patching process.
Put the site back online again.

That’s the abridged version. See the UPGRADE.txt file if you’ve got time to burn.

All of which means that, unless you spend your life administering Drupal websites professionally, your site is probably going to lag behind the latest version for a while until you can summon up the energy to go through the pain of updating it. I’d go as far as to guess that most Drupal websites are not up to date most of the time.

Even with resources such as Drush and Bernhard Fürst’s version patches, the whole process is orders of magnitude more painful than I’d like it to be.

The final nail in the coffin

So what is the tipping point? What has finally nudged me to abandon Drupal for good?

It’s this security advisory (which I somehow missed at the time) and its follow-up (which I almost missed too).

Basically, if you were using Drupal and hadn’t updated the software then your site was wide open to hackers and there was a good chance that it had already been compromised by a wide-scale automated attack. Why constantly fight to keep installing security patch on security patch when you can just serve it all up as static content?

What’s good about Jekyll?

So what is Jekyll?

Jekyll essentially knows about markdown and blogs, and how to turn one into the other. It’s the engine that powers Github Pages, so it’s got a good pedigree.

Once it’s installed, you can just start writing posts as markdown files. You can set Jekyll’s local server running and watching for changes and view the blog in a web browser pointed at localhost (http://localhost:4000 by default).

From there it’s just a matter of copying the static content to your server. I’m using site44 so deployment is as simple as copying it from one Dropbox folder to another and waiting for it to sync.

What else did I consider?

Octopress

Octopress (http://octopress.org/) is built on top of Jekyll, so one is worthy of consideration if you’re thinking of using the other. In theory Octopress simplifies working with Jekyll by providing various templates and helper scripts. However, after reading various posts online from people switching from Octopress back to plain Jekyll – for less complexity and more control – I decided to plump for Jeckyll on its own.

If you’ve written HTML and CSS in the past then editing the HTML templates and sass/scss styles used by Jekyll is actually quite pleasant, so I’m not sure Octopress would add much apart from complexity.

SquareSpace

SquareSpace would take away all the pain of administering you own server by providing that for you. I’ve used it before for various projects.

It is a fine option if you don’t already have hosting space that you’re paying for. If you do, then it’s just a little too expensive, especially if you don’t need more than one user to be able to post updates.

They’d do well to have an inexpensive cut-down option with very low storage and bandwidth to encourage people into their ecosystem.

Getting up and running with Jekyll on Windows

Jekyll’s really easy to use on Linux or a Mac. On Windows it’s a bit more fiddly to set up: some combinations of the various components work better than others. There are basically two ways to go: follow the excellent instructions here or download the whole lot pre-configured from PortableJekyll. I went the latter route.

Go to https://github.com/madhur/PortableJekyll and fork it.

Open Github for Windows and click +, Clone, Portable Jekyll, Tick.

Select a folder to clone it into. I chose the folder Software in my Dropbox. t takes quite a while to fetch the content, so go off and do something else while you’re waiting. You end up with a PortableJekyll folder beneath the folder you selected. (You could just download the latest snapshot from GitHub and unpack it onto disk without the repository, but doing it via Git should make updating to future versions slightly easier.)

Creating a blog

Creating an empty Jekyll project

In a console, change to the parent folder of where you want to create your site.

Run the setpath.cmd file from the folder you saved Portable Jekyll into:

c:\Users\Matthew\Dropbox\Software\PortableJekyll\setpath.cmd

This adds the components of Portable Jekyll to the path.

Now use the jekyll new command to construct an empty Jekyll project:

c:\Users\Matthew\Dropbox\Web>jekyll new MatthewStrawbridge.com

New jekyll site installed in c:/Users/Matthew/Dropbox/Web/MatthewStrawbridge.com.

This creates various files and folders to form a very simple website.

Now is a good opportunity to add the empty site as a baseline to Git:

cd MatthewStrawbridge.com

git init

git add .

git commit -m "baseline"

Setting up syntax highlighting

Jekyll ships with a couple of syntax highlighters:

Pygments is the default. It’s written in Python and supports syntax highlighting in lots of languages.
Rouge is written in Ruby and supports fewer languages, but still has all the ones I’m likely to need.

I had issues getting Pygments to run so I added the following to my site’s _config.yml file to switch it to use Rouge:

highlighter: rouge

Building and viewing the site

Now we can try building the new empty site.

jekyll build

If you don’t get any red text output to the console then it has probably worked.

So we can now try the site:

jekyll serve --force-polling

(You might need to allow the Windows firewall to let Ruby do its thing, if you get asked.)

It looks like you need to use the --force_polling option on Windows. Without this, once watch was triggered by a change, it tried to regenerate the whole site once per second even when there were no further changes! But with this option it sits there quite happily, only springing into action when a genuine change has been made.

This starts a local webserver and watches your project for changes. To see the site, just go to http://localhost:4000/ in a web browser.

You should see something like this:

Clicking on the title of the default post which has been created for you gives you an idea of the format you can expect, as well as providing some basic instructions about what lives where:

This is a good point to tweak the various settings in _config.xml to make the site feel more your own. The server will continue to run in the background so once you save the file, give it a second or two and you can reload it in your browser.

You might need to kill the Jekyll server and restart it to detect that the config file has changed.

Adding posts

Adding a new post is as simple as creating a new markdown (or HTML if you prefer) file in the _posts directory. If you use the default naming convention of yyyy-mm-dd-title then you don’t have to specify the post’s date and title in the metadata section at the top of the file.

As an example, the top of the markdown file for this post looks like this:

---

layout:     post

title:      "Migrating from Drupal to Jekyll"

date:       2014-11-08 23:30:00

published:  true

---


# What's this blog post about?


The content of this blog has been migrated from one platform to
another several times now. Back in the day I had an account on
[Blogger](http://philoxenic.blogspot.co.uk/). It looks like it's still there,
which probably isn't doing wonders for the SEO on this site with the copied
content.

You can read more about Markdown formatting here.

Summing up

So that’s it. I’ve moved from a fairly heavyweight content management system to lightweight HTML. It’s just as easy to write posts in Markdown as it was to use Drupal’s editing system, and publishing new posts is simply a matter of running a script to copy the generated site from one part of my Dropbox hierarchy to another.

If you’re looking for a way to publish a blog or other content as HTML but prefer writing in Markdown then give Jekyll a try.

Smarter Classes with Less Code (Part 2 of 2)

Sun, 21 Sep 2014 21:14:00 +0100

Last time we took a step towards adding useful functionality to C# types to automatically add some of the features you get for free in functional languages, such as immutability and equality comparisons based on properties.

This time we’ll look at how to use PostSharp to inject the functionality we’re interested in based on custom attributes. Not only will that be more efficient at runtime than relying on .NET reflection, it will allow us to derive our types from others if we wish.

PostSharp immutable threading pattern

As I was trying to write the code for this article, I came across this article: New in PostSharp 3.2: Immutable and Freezable Done Well. This discusses some new attributes available only in recent versions of PostSharp, so you need to install it via NuGet with the “Include Prerelease” option selected. You also need to install the pre-release version of the PostSharp Threading Pattern Library.

This means you can define the value class with a simple [Immutable] attribute like this:

using PostSharp.Patterns.Threading;

namespace ValueClasses
{
    [Immutable]
    public class Item
    {
        public int Id { get; private set; }
        public string Description { get; private set; }

        public Item(int id, string description)
        {
            Id = id;
            Description = description;
        }

        public void ChangeDescription()
        {
            Description = "changed";
        }
    }
}

And then if we run a program like this:

using System;
using ValueClasses;

namespace DDD2
{
    class Program
    {
        static void Main()
        {
            var item = new Item(123, "My Description");
            Console.WriteLine(item);
            item.ChangeDescription();
            Console.WriteLine(item);
        }
    }
}

… then we get an exception thrown from the ChangeDescription call.

That’s pretty awesome! PostSharp is preventing the class’s own method from changing its property’s backing field after construction. That’s a level of protection (and bug prevention) you can’t get through private setters.

Decompiling the resultant assembly with JetBrains’ free stand-alone tool dotPeek shows just how much extra code PostSharp has injected to get this to work:

using PostSharp.Aspects;
using PostSharp.Aspects.Advices;
using PostSharp.Aspects.Internals;
using PostSharp.Extensibility;
using PostSharp.Patterns.Model;
using PostSharp.Patterns.Threading;
using PostSharp.Reflection;
using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;

namespace ValueClasses
{
  [HasInheritedAttribute(new long[] {})]
  [Immutable(AttributeId = 881225518374429271L, AttributePriority = 268435456)]
  [HasOnInstanceInitialized]
  public class Item : IAttachable, IImmutable
  {
    private int \u003CId\u003Ek__OriginalField;
    private string \u003CDescription\u003Ek__OriginalField;
    [NonSerialized]
    private ImmutableAttribute \u003C\u003Ez__aspect0;
    [NonSerialized]
    private AggregatableAttribute \u003C\u003Ez__aspect1;

    public int Id
    {
      get
      {
        return this.get_\u003CId\u003Ek__BackingField();
      }
      private set
      {
        this.set_\u003CId\u003Ek__BackingField(value);
      }
    }

    public string Description
    {
      get
      {
        return this.get_\u003CDescription\u003Ek__BackingField();
      }
      private set
      {
        this.set_\u003CDescription\u003Ek__BackingField(value);
      }
    }

    [CompilerGenerated]
    private int \u003CId\u003Ek__BackingField
    {
      [DebuggerNonUserCode] get
      {
        LocationInterceptionArgsImpl<int> interceptionArgsImpl = new LocationInterceptionArgsImpl<int>((object) this, Arguments.Empty);
        interceptionArgsImpl.DeclarationIdentifier = new DeclarationIdentifier(881225515594350592L);
        interceptionArgsImpl.TypedBinding = (LocationBinding<int>) Item.\u003CId\u003Ec__Binding0.singleton;
        this.\u003C\u003Ez__aspect0.OnFieldGet((LocationInterceptionArgs) interceptionArgsImpl);
        return interceptionArgsImpl.TypedValue;
      }
      [DebuggerNonUserCode] set
      {
        LocationInterceptionArgsImpl<int> interceptionArgsImpl = new LocationInterceptionArgsImpl<int>((object) this, Arguments.Empty);
        interceptionArgsImpl.DeclarationIdentifier = new DeclarationIdentifier(881225515594350592L);
        interceptionArgsImpl.TypedBinding = (LocationBinding<int>) Item.\u003CId\u003Ec__Binding.singleton;
        interceptionArgsImpl.TypedValue = value;
        this.\u003C\u003Ez__aspect0.OnFieldSet((LocationInterceptionArgs) interceptionArgsImpl);
      }
    }

    [CompilerGenerated]
    private string \u003CDescription\u003Ek__BackingField
    {
      [DebuggerNonUserCode] get
      {
        LocationInterceptionArgsImpl<string> interceptionArgsImpl = new LocationInterceptionArgsImpl<string>((object) this, Arguments.Empty);
        interceptionArgsImpl.DeclarationIdentifier = new DeclarationIdentifier(881225515594350593L);
        interceptionArgsImpl.TypedBinding = (LocationBinding<string>) Item.\u003CDescription\u003Ec__Binding0.singleton;
        this.\u003C\u003Ez__aspect0.OnFieldGet((LocationInterceptionArgs) interceptionArgsImpl);
        return interceptionArgsImpl.TypedValue;
      }
      [DebuggerNonUserCode] set
      {
        LocationInterceptionArgsImpl<string> interceptionArgsImpl = new LocationInterceptionArgsImpl<string>((object) this, Arguments.Empty);
        interceptionArgsImpl.DeclarationIdentifier = new DeclarationIdentifier(881225515594350593L);
        interceptionArgsImpl.TypedBinding = (LocationBinding<string>) Item.\u003CDescription\u003Ec__Binding.singleton;
        interceptionArgsImpl.TypedValue = value;
        this.\u003C\u003Ez__aspect0.OnFieldSet((LocationInterceptionArgs) interceptionArgsImpl);
      }
    }

    protected Item(int id, string description, ConstructorDepth __depth)
    {
      // ISSUE: reference to a compiler-generated method
      this.\u003C\u003Ez__InitializeAspects(AspectInitializationReason.Constructor);
      this.Id = id;
      this.Description = description;
      if (!__depth.IsZero)
        return;
      // ISSUE: reference to a compiler-generated method
      this.OnInstanceInitialized();
    }

    public void ChangeDescription()
    {
      this.Description = "changed";
    }

    protected virtual void OnChildDetached(object child, ChildInfo childInfo)
    {
      this.\u003C\u003Ez__aspect1.OnChildDetached(child, childInfo);
    }

    protected virtual void OnParentChanged()
    {
      this.\u003C\u003Ez__aspect0.OnParentChanged();
    }

    protected virtual void OnChildAttached(object child, ChildInfo childInfo)
    {
      this.\u003C\u003Ez__aspect0.OnChildAttached(child, childInfo);
    }

    protected virtual void OnAncestorChanged(AncestorChangedEventArgs args)
    {
      this.\u003C\u003Ez__aspect1.OnAncestorChanged(args);
    }

    protected virtual bool VisitChildren(ChildVisitor visitor, ChildVisitorOptions options)
    {
      return this.\u003C\u003Ez__aspect1.VisitChildren(visitor, options);
    }

    private void \u003COnChildAttached\u003Ez__(object child, ChildInfo childInfo)
    {
      this.\u003C\u003Ez__aspect1.OnChildAttached(child, childInfo);
    }

    private void \u003COnParentChanged\u003Ez__()
    {
      this.\u003C\u003Ez__aspect1.OnParentChanged();
    }
  }
}

So, yeah… you wouldn’t really want to write that lot by hand to achieve the same thing.

Other aspects

As well as the [Immutable] aspect, there’s also a [Freezable] model. This is similar but you start with a fully mutable object that you can make changes to and at some point you call its Freeze() method and after that you can no longer change it.

Adding Equals and GetHashCode

So that’s a summary of the type of thing PostSharp can do. What about adding Equals and GetHashCode like the previous article?

I couldn’t work out how to do this myself so I asked the question on Stack Overflow.

Here’s some code based on the answer I got there:

using System;
using System.Collections.Generic;
using System.Reflection;
using PostSharp.Aspects;
using PostSharp.Aspects.Advices;
using PostSharp.Reflection;
using PostSharp.Serialization;

namespace ValueClasses
{
    [PSerializable]
    class AutoEqAttribute : InstanceLevelAspect, IAdviceProvider
    {
        /// <summary>
        /// The fields from the class this attribute has been added to
        /// </summary>
        /// <remarks>
        /// Automatically populated by PostSharp with the fields identified by
        /// <see cref="ProvideAdvices"/>
        /// </remarks>
        public List<ILocationBinding> Fields;

        /////////
        // Equals

        // Grab a reference to the base class's Equals method

        [ImportMember("Equals", IsRequired = true, Order = ImportMemberOrder.BeforeIntroductions)]
        public Func<object, bool> EqualsBaseMethod;

        // Add a new Equals method to the class which this attribute has been added to

        [IntroduceMember(IsVirtual = true, OverrideAction = MemberOverrideAction.OverrideOrFail)]
        public new bool Equals(object other)
        {
            // TODO: Define a smarter way to determine if base.Equals should be invoked.
            if (EqualsBaseMethod.Method.DeclaringType != typeof(object))
            {
                if (!EqualsBaseMethod(other))
                    return false;
            }

            object instance = Instance;
            foreach (ILocationBinding binding in Fields)
            {
                // The following code is inefficient because it boxes all fields.
                // There is currently no workaround.
                object thisFieldValue = binding.GetValue(ref instance, Arguments.Empty);
                object otherFieldValue = binding.GetValue(ref other, Arguments.Empty);

                if (!object.Equals(thisFieldValue, otherFieldValue))
                    return false;
            }

            return true;
        }

        //////////////
        // GetHashCode

        [IntroduceMember(IsVirtual = true, OverrideAction = MemberOverrideAction.OverrideOrFail)]
        public new int GetHashCode()
        {
            object instance = Instance;

            unchecked
            {
                int hashcode = 11;
                foreach (ILocationBinding binding in Fields)
                {
                    object thisFieldValue = binding.GetValue(ref instance, Arguments.Empty);
                    int thisHash = thisFieldValue == null ? 0 : thisFieldValue.GetHashCode();
                    hashcode = (hashcode * 397) ^ thisHash;
                }

                return hashcode;
            }
        }

        /// <summary>
        /// Provide advices for setting up <see cref="Fields"/>
        /// </summary>
        /// <param name="targetElement">The type to which this attribute has been added</param>
        /// <returns>A sequence of advice instances</returns>
        public IEnumerable<AdviceInstance> ProvideAdvices(object targetElement)
        {
            var targetType = (Type)targetElement;
            FieldInfo bindingField = GetType().GetField("Fields");

            const BindingFlags bindingFlags =
                BindingFlags.DeclaredOnly |
                BindingFlags.Instance |
                BindingFlags.Public |
                BindingFlags.NonPublic;

            FieldInfo[] fields = targetType.GetFields(bindingFlags);

            foreach (FieldInfo field in fields)
            {
                yield return new ImportLocationAdviceInstance(bindingField, new LocationInfo(field));
            }
        }
    }
}

Then you can just add a [AutoEq] attribute to your simple class and get Equals and GetHashCode automatically implemented.

Other options

T4 templates

While writing this article I read this Stack Overflow question that suggests doing the same thing by using T4 templates (specifically the T4 C# Constructor Generator project) to add this functionality via partial classes.

Resharper

In this particular example, perhaps the most pragmatic option is to use Resharper’s menu option Resharper | Edit | Generate Code | Equality Members. You can just pick which fields and properties should contribute, say whether you want to implement IEquatable<T> as well, and leave Resharper to generate the boilerplate code for you.

All three options are worth knowing about and have their strengths and weaknesses. Postsharp is good for adding cross-cutting aspects to your code, logging being the canonical example. T4 templates look to be useful for automatically generating textual content, including source code, but aren’t widely used. Finally, the fact that Resharper could do the heavy lifting all along is a reminder that it’s always worth investing the time to understand the capabilities of the tools you already have before you start looking further afield.

Smarter Classes with Less Code (Part 1 of 2)

Sun, 21 Sep 2014 00:00:00 +0100

Last Saturday I went to the Developer Developer Developer conference in Cambridge (DDD East Anglia), which was both very good and free to attend.

As often happens at such conferences, it’s a throw-away remark or a discussion in the bar afterwards that has the most lasting impact rather than the subjects of the talks themselves.

In this case it was the talk by Alessandro Melchiori entitled Functional DDD that created the spark.

The traditional hand-coded approach

So, for example if you have some simple code like this, with an Item class that has a couple of properties, then its ToString isn’t very helpful:

using System;

namespace DDD1
{
    public class Item
    {
        public int Id { get; set; }
        public string Description { get; set; }
    }

    class Program
    {
        static void Main()
        {
            var item = new Item { Id = 123, Description = "My item" };
            Console.WriteLine(item);
        }
    }
}

The output you get is as follows:

DDD1.Item

So all you know is that you’re dealing with an Item, but you know nothing about its contents.

Traditionally you’d have to code up an override of ToString that knows about the properties of Item, something like this:

        public override string ToString()
        {
            return string.Format("Item {{ Id = {0}, Description = {1} }}", Id, Description);
        }

Which would give you this more useful output:

Item { Id = 123, Description = My item }

The JSON trick

In his talk, Alessandro Melchiori happened to mention using the Json.NET library’s JsonConvert class as an easy way of implementing ToString in your classes.

You can install Json.NET into your solution in a few seconds using NuGet. Then it’s just a case of adding a using statement and changing your ToString to use JsonConvert.SerializeObject:

using System;
using Newtonsoft.Json;

namespace DDD1
{
    public class Item
    {
        public int Id { get; set; }
        public string Description { get; set; }

        public override string ToString()
        {
            return JsonConvert.SerializeObject(this);
        }
    }

    class Program
    {
        static void Main()
        {
            var item = new Item { Id = 123, Description = "My item" };
            Console.WriteLine(item);
        }
    }
}

This gives the following output:

{"Id":123,"Description":"My item"}

Now, that might not seem like a big deal.

The key point is that this one line of code inside the ToString will work however many properties we’ve got in our class, and will automatically be kept in sync as we add or remove properties. For most types it can provide really useful output for minimal coding effort.

Furthermore, if we stick that functionality in a base class then all you need to do to add this useful behaviour is derive from that class.

Adding a ValueBase class

If we’re creating a base class we can dispense with the Json.NET trick and use standard reflection to do clever things with our properties.

The following example makes it easy to create immutable types (notice that Item’s properties are read-only) by providing a generic constructor. It also includes a ToString and some sensible default implementations of Equals and GetHashCode too.

using System;
using System.Linq;
using System.Reflection;
using System.Text;

namespace DDD1
{
    /// <summary>
    /// Base class for simple classes with properties
    /// </summary>
    public abstract class ValueBase
    {
        public override string ToString()
        {
            Type type = GetType();

            var result = new StringBuilder();
            result.AppendFormat("{0}:\n", type.Name);

            var props = type.GetProperties();
            foreach (var prop in props)
            {
                result.AppendFormat("    {0}: {1}\n", prop.Name, prop.GetValue(this));
            }

            return result.ToString();
        }

        public override bool Equals(object other)
        {
            Type type = GetType();

            if (other == null || !type.IsInstanceOfType(other))
            {
                return false;
            }

            foreach (var prop in type.GetProperties())
            {
                var thisProp = prop.GetValue(this);
                var thatProp = prop.GetValue(other);

                if (!thisProp.Equals(thatProp))
                {
                    return false;
                }
            }

            return true;
        }

        public override int GetHashCode()
        {
            Type type = GetType();
            var props = type.GetProperties();
            PropertyInfo firstProp = props.FirstOrDefault();
            return firstProp == null ? 0 : firstProp.GetValue(this).GetHashCode();
        }

        protected ValueBase(object properties)
        {
            Type type = GetType();
            var props = type.GetProperties();

            var otherProps = properties.GetType().GetProperties();

            foreach (PropertyInfo otherProp in otherProps)
            {
                props.Single(p => p.Name.Equals(otherProp.Name)).SetMethod.Invoke(this, new[] { otherProp.GetValue(properties) });
            }
        }
    }

    public class Item : ValueBase
    {
        public int Id { get; private set; }
        public string Description { get; private set; }

        public Item(object properties) : base(properties) { }
    }

    class Program
    {
        static void Main()
        {
            var item = new Item(new { Id = 123, Description = "My item" });
            Console.WriteLine(item);
        }
    }
}

Notice how simple the Item class is. Just by deriving it from the base class we’ve got a nice immutable package of properties with useful ToString output and sane equality comparisons (by property values rather than object references). If you had lots of small types like this then it would be easy to add this behaviour without having to write lots of boilerplate code each time.

If this all sounds familiar, it’s the kind of thing you typically get for free in functional languages such as Haskell and F#. Once you create an object it’s safely immutable and equality is automatically based on its content rather than its location in memory. That’s often useful behaviour in C# too, but there are normally lots of hoops to jump through to get there.

There are a couple of downsides to the ValueBase approach, however. First, we’re using reflection to save us coding effort, so the resulting code will be slower than the custom version you’d normally have. Second, we still have to derive from a base class and hit problems because C# doesn’t allow multiple inheritance. We can overcome both issues by switching to a solution using aspect-oriented programming (AOP) with PostSharp. I’ll cover that in the next article.

Matthew Strawbridge's weblog

Serving Python on AWS Lambda with Zappa

What is Zappa?

Okay, but this only works for simple things, right?

Setting up Zappa in practice

Package zip sizes

Minimal AWS permissions set

Keeping warm

Example service

Conclusion

Frameworks

Creating a balanced investment portfolio with skfolio in Python

Motivation

Python libraries

skfolio

yfinance

Matplotlib

Declaring an asset universe

Setting constraints

Optimising a portfolio with skfolio

Output

Historical price data

Correlation matrix

Cumulative returns

Portfolio composition

Full script

Edits:

Ghostly paragraphs in MkDocs

Ghostly goings-on

Background

The set-up

Reproducing the problem

Does it matter?

So, what’s going on?

Summary

Python logging: basic, better and best

Basic: print statements

Better: logging

Base logger

A note about message construction

Custom logger

Best: structlog

Binding values

Exceptions

Typical structlog config

A note about async logging

Other options

Loguru

Logbook

Sentry

python-json-logger

Wrap-up

Cloudflare DNS and Site44

The problem: could not load this blog without the www. prefix

The solution: adding a CloudFlare redirect rule

1. A CNAME link from the custom domain to Site44

2. An A record entry for the top-level host

3. Create the redirection rule

Conclusion

Thoughts on writing code well

About this post

Test-driven design (TDD)

Degrees of hatred and fandom

Definitions

Killing disease at the point of contraction

How unit tests stop bugs

Avoiding bugs altogether where you can

If there are bugs, finding them early

Simplicity

The first software I installed on my new Windows laptop

1. Chrome

2. Paint.NET

3. Dropbox

4. Vim

5. ConEmu

6. Inconsolata font

7. LinqPad

8. Beyond Compare

9. Directory Opus

10. Scrivener

1. A `CNAME` link from the custom domain to Site44

2. An `A` record entry for the top-level host