Month: July 2021

[AdSense-A]

When reading technical blogpost around the web, you might have noticed that code highlighting is not always perfect.

Shiki is the code highlighter that uses the textmate parser VSCode uses under the hood. The code highlighting it provides is near perfect, even when using modern syntax. It supports 100+ languages (via our package Blade is supported too), and all VS Code themes.

I'm proud to announce that we have released three new Spatie packages that make it easy to use Shiki in your PHP projects:

• shiki-php: makes it easy to call Shiki from PHP to highlight a given code snippet
• commonmark-shiki-highlighter: allows commonmark to highlight all code snippets in a markdown fragment
• laravel-markdown: a batteries included Laravel package that offers a Blade component to easily render Markdown with highlighted code snippets and a class to render Markdown manually.

We're already using this package to render all our documentation pages, our guidelines, and this very blog you are reading.

A little bit of history

On this very blog, spatie.be and related sites, we have relied on our trusty commonmark-highlighter package for the past couple of years.

This package offers a renderer class that can be used with the league's popular commonmark package. This renderer uses highlight.php to render code blocks in Markdown. Highlight.php is inspired by hightlight.js. While the highlighting provides by highlight.php/js is pretty good, it's not perfect.

A couple of months ago, Miguel Piedrafita blogged about highlighting using Shiki. Shiki is the code renderer behind VSCode. Its highlighting is near perfect, and it can even handle modern PHP Syntax. Miguel's blog post mentioned how you could use Shiki in a JS / Node environment. My colleague Rias used that blog post as inspiration to toy with Shiki and bringing its magic to the PHP world.

Introducing Shiki PHP

The first package we released, spatie/shiki-php, allows you to highlight any given code snippet.

Here's a good first example where we are going to highlight a simple PHP script. Using Shiki is simple, you pass it some code, the language of the code you are passing and one of the many available themes.

use SpatieShikiPhpShiki;

Shiki::highlight(
    code: '<?php echo "Hello World"; ?>',
    language: 'php',
    theme: 'github-light',
);

This is the html that Shiki will be returned:

<pre class="shiki" style="background-color: #2e3440ff"><code><span class="line"><span style="color: #81A1C1">&lt;?</span><span style="color: #D8DEE9FF">php </span><span style="color: #81A1C1">echo</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">&quot;</span><span style="color: #A3BE8C">Hello World</span><span style="color: #ECEFF4">&quot;</span><span style="color: #81A1C1">;</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">?&gt;</span></span></code></pre>

In the browser, this will be displayed as:

<?php echo "Hello World"; ?>

This very blog you are reading uses Shiki to highlight code snippets. Let's try a few more advanced examples. Here's a very meta example in which we highlight the code of the Shiki class from the package.

namespace SpatieShikiPhp;

use SymfonyComponentProcessExceptionProcessFailedException;
use SymfonyComponentProcessExecutableFinder;
use SymfonyComponentProcessProcess;

class Shiki
{
    public static function highlight(
        string $code,
        string $language = 'php',
        string $theme = 'nord',
        array $highlightLines = [],
        array $addLines = [],
        array $deleteLines = [],
        array $focusLines = [],
    ): string {
        return (new static())->highlightCode($code, $language, $theme, [
            'highlightLines' => $highlightLines,
            'addLines' => $addLines,
            'deleteLines' => $deleteLines,
            'focusLines' => $focusLines,
        ]);
    }

    public function getAvailableLanguages(): array
    {
        $shikiResult = $this->callShiki('languages');

        $languageProperties = json_decode($shikiResult, true);

        $languages = array_map(
            fn ($properties) => $properties['id'],
            $languageProperties
        );

        sort($languages);

        return $languages;
    }

    public function __construct(
        protected string $defaultTheme = 'nord'
    ) {
    }

    public function getAvailableThemes(): array
    {
        $shikiResult = $this->callShiki('themes');

        return json_decode($shikiResult, true);
    }

    public function languageIsAvailable(string $language): bool
    {
        return in_array($language, $this->getAvailableLanguages());
    }

    public function themeIsAvailable(string $theme): bool
    {
        return in_array($theme, $this->getAvailableThemes());
    }

    public function highlightCode(string $code, string $language, ?string $theme = null, ?array $options = []): string
    {
        $theme = $theme ?? $this->defaultTheme;

        return $this->callShiki($code, $language, $theme, $options);
    }

    protected function callShiki(...$arguments): string
    {
        $command = [
            (new ExecutableFinder)->find('node', 'node'),
            'shiki.js',
            json_encode(array_values($arguments)),
        ];

        $process = new Process(
            command: $command,
            cwd: realpath(__DIR__ . '/../bin'),
            timeout: null,
        );

        $process->run();

        if (! $process->isSuccessful()) {
            throw new ProcessFailedException($process);
        }

        return $process->getOutput();
    }
}

Notice that the highlighting in the code above is perfect. Even the features that were added in PHP 8, named arguments, are highlighted correctly.

Let's now try to highlight the Blade view used to render this very page you are reading.

<x-app-layout :title="$post->title">
    <x-ad/>

    <x-post-header :post="$post" class="mb-8">

        {!! $post->html !!}

        @unless($post->isTweet())
            @if($post->external_url)
                <p class="mt-6">
                    <a href="{{ $post->external_url }}">
                        Read more</a>
                    <span class="text-xs text-gray-700">[{{ $post->external_url_host }}]</span>
                </p>
            @endif
        @endunless
    </x-post-header>

    @include('front.newsletter.partials.block', [
        'class' => 'mb-8',
    ])

    <div class="mb-8">
        @include('front.posts.partials.comments')
    </div>

    <x-slot name="seo">
        <meta property="og:title" content="{{ $post->title }} | freek.dev"/>
        <meta property="og:description" content="{{ $post->plain_text_excerpt }}"/>
        <meta name="og:image" content="{{ url($post->getFirstMediaUrl('ogImage')) }}"/>

        @foreach($post->tags as $tag)
            <meta property="article:tag" content="{{ $tag->name }}"/>
        @endforeach

        <meta property="article:published_time" content="{{ optional($post->publish_date)->toIso8601String() }}"/>
        <meta property="og:updated_time" content="{{ $post->updated_at->toIso8601String() }}"/>
        <meta name="twitter:card" content="summary_large_image"/>
        <meta name="twitter:description" content="{{ $post->plain_text_excerpt }}"/>
        <meta name="twitter:title" content="{{ $post->title }} | freek.dev"/>
        <meta name="twitter:site" content="@freekmurze"/>
        <meta name="twitter:image" content="{{ url($post->getFirstMediaUrl('ogImage')) }}"/>
        <meta name="twitter:creator" content="@freekmurze"/>
    </x-slot>
</x-app-layout>

Again, perfect! Finally, let's try a bit of Vue taken from medialibrary.pro.

<template>
    <div>
        <heading class="mb-6">Generate Newsletter</heading>

        <card class="py-3 px-6">
            <div class="flex border-b border-40">
                <div class="w-1/4 py-4">
                    <slot>
                        <h4 class="font-normal text-80">
                            Edition number
                        </h4>
                    </slot>
                </div>
                <div class="w-3/4 py-4">
                    <slot name="value">
                        <input
                          v-model="form.editionNumber"
                          type="text"
                          class="w-full form-control form-input form-input-bordered flatpickr-input active"
                        />

                    </slot>
                </div>
            </div>

            <div class="flex border-b border-40">
                <div class="w-1/4 py-4">
                    <slot>
                        <h4 class="font-normal text-80">
                            Start date
                        </h4>
                    </slot>
                </div>
                <div class="w-3/4 py-4">
                    <slot name="value">
                        <input
                          v-model="form.startDate"
                          type="text"
                          placeholder="2018-01-01"
                          class="w-full form-control form-input form-input-bordered flatpickr-input active"
                        />

                    </slot>
                </div>
            </div>

            <div class="flex border-b border-40">
                <div class="w-1/4 py-4">
                    <slot>
                        <h4 class="font-normal text-80">
                            End date
                        </h4>
                    </slot>
                </div>
                <div class="w-3/4 py-4">
                    <slot name="value">
                        <input
                          v-model="form.endDate"
                          type="text"
                          placeholder="2018-01-01"
                          class="w-full form-control form-input form-input-bordered flatpickr-input active"
                        />
                    </slot>
                </div>
            </div>

            <div class="flex border-b border-40" v-if="newsletterHtml !== ''">
                <div class="w-1/4 py-4">
                    <slot>
                        <h4 class="font-normal text-80">
                            Newsletter html
                        </h4>
                    </slot>
                </div>
                <div class="w-3/4 py-4">
                    <slot name="value">
                        <textarea
                          style="height: 270px"
                          v-model="newsletterHtml"
                          type="text"
                          class="w-full form-control form-input form-input-bordered flatpickr-input active"
                        ></textarea>
                    </slot>
                </div>
            </div>

        </card>
        <div class="bg-30 flex px-8 py-4">
            <button
              @click="generateNewsletter"
              type="button"
              class="ml-auto btn btn-default btn-primary mr-3">
            Generate newsletter
        </button></div>
    </div>
</template>

<script>
export default {
    data() {
        return {
            form: {
                editionNumber: null,
                startDate: null,
                endDate: null,
            },

            newsletterHtml: '',
        };
    },

    methods: {
        async generateNewsletter() {
            let response = await window.axios.post(
                '/nova-vendor/freekmurze/generate-newsletter',
                this.form
            );

            this.newsletterHtml = response.data;
        },
    },
};
</script>

The package has few additional features. Head over to the spatie/shiki-php readme to learn more.

Highlighting code snippets in Markdown

Using the spatie/shiki-php package, you can highlight a given single code snippet. In most cases, you'll not use it directly. Most blogs and documentation pages use Markdown to store content. That Markdown can contain code snippets, like this page from our docs.

To render Markdown to HTML, the popular league/commonmark package can be used. One of the nice things about that package is that it is very extensible.

Our second new package, spatie/commonmark-shiki-highlighter, contains an extension for the commonmark package that will highlight all code snippets in a markdown snippet using Shiki PHP.

Here's how you can create a function that can converts Markdown to HTML with all code snippets highlighted. Inside the function will create a new CommonMarkConverter that uses the HighlightCodeExtension provided by our package.

use LeagueCommonMarkCommonMarkConverter;
use LeagueCommonMarkEnvironment;
use SpatieCommonMarkShikiHighlighterHighlightCodeExtension;

function convertToHtml(string $markdown, string $theme = 'github-light'): string
{
    $environment = Environment::createCommonMarkEnvironment()
        ->addExtension(new HighlightCodeExtension($theme));

    $commonMarkConverter = new CommonMarkConverter(environment: $environment);

    return $commonMarkConverter->convertToHtml($markdown);
}

With that function setup, you can easily convert Markdown to HTML anywhere in your project.

$markdown = <<<MD
# A code snippet

Here is a code snippet that will be highlighted correctly.

```php
echo 'Hello world';
```
MD;

$html = convertToHtml($markdown)

The package also has support for [marking lines as highlighted, added, deleted and focused].

To highlight a line, simply add the line number that you want to highlight between brackets after the language of a code snippet, for example:

```php{2}

Here's a bit of code in which we highlighted the second line.

function myFunction() {
	return 'this line will be highlighted';
}

You can also mark lines as added en deleted by prefix a line with + or -. This piece of Markdown...

```php
<?php
echo "This line is marked as added";
echo "This line is marked as deleted";
```

... will be rendered as:

<?php
echo "This line is marked as added";
echo "This line is marked as deleted";

These highlighting features work because Shiki will apply some CSS classes to the highlighted lines. You should write a bit of CSS of your own that determines how highlighted lines look like. Here are the classes I use on this very blog (my colleague Rias originally wrote this CSS).

.shiki .highlight {
    background-color: hsl(197, 88%, 94%);
    padding: 3px 0;
}

.shiki .add {
    background-color: hsl(136, 100%, 96%);
    padding: 3px 0;
}

.shiki .del {
    background-color: hsl(354, 100%, 96%);
    padding: 3px 0;
}

.shiki.focus .line:not(.focus) {
    transition: all 250ms;
    filter: blur(2px);
}

.shiki.focus:hover .line {
    transition: all 250ms;
    filter: blur(0);
}

Did you notice those last two CSS classes. They are used to style focussed code. It's pretty cool that Shiki supports this. Let's see an example of that in action.

To focus on particular pieces of code, simply put the line numbers in the second pair of brackets.

```css{}{16-24}

Let's again go meta a focus on the CSS bits that enable focus.

.shiki .highlight {
    background-color: hsl(197, 88%, 94%);
    padding: 3px 0;
}

.shiki .add {
    background-color: hsl(136, 100%, 96%);
    padding: 3px 0;
}

.shiki .del {
    background-color: hsl(354, 100%, 96%);
    padding: 3px 0;
}

.shiki.focus .line:not(.focus) {
    transition: all 250ms;
    filter: blur(2px);
}

.shiki.focus:hover .line {
    transition: all 250ms;
    filter: blur(0);
}

To know more about highlighting, focussing, ... head over to the docs on commonmark-shiki-highlighting.

A Blade component to render Markdown with highlighted code

Using our commonmark shiki extension is not that hard, but for Laravel apps, we can make the usage even easier.

Let's look at the third package we released: spatie/laravel-markdown. This package contains:

• a Blade component that can render markdown
• a highly configurable class that you can use to render Markdown

Let's start with an example of the provided x-markdown Blade component. The component can convert this chunk of Markdown...

<x-markdown>
This is a [link to our website](https://spatie.be)

```php
echo 'Hello world';
```
</x-markdown>

... to this chunk of HTML:

<div>
    <h1 id="my-title">My title</h1>
    <p>This is a <a href="https://spatie.be">link to our website</a></p>
    <pre class="shiki" style="background-color: #fff"><code><span class="line"><span
        style="color: #005CC5">echo</span><span style="color: #24292E"> </span><span style="color: #032F62">'Hello world'</span><span
        style="color: #24292E">;</span></span>
<span class="line"></span></code></pre>
</div>

Which will be displayed in the browser as:

This is a link to our website

echo 'Hello world';

Of course, all code blocks are highlighted using Shiki.

Using the x-markdown component is the easiest way to render Markdown in a Laravel app. Previously, I've already stated that rendering markdown using Shiki can be resource-intensive. You'll be happy to know that the x-markdown Blade component has built-in caching. A unique piece of Markdown will only get rendered once, so you'll only have the performance hit the first time.

If you want to take care of the markdown rendering yourself, without using the Blade component, you can use the MarkdownRenderer class that ships with the package. Here's an example of how you can use it.

$html = app(SpatieLaravelMarkdownMarkdownRenderer::class)->toHtml($markdown);

Of course, the MarkdownRenderer will highlight code snippets in the markdown, and cache the results. It's also easy to customise the rendering:

$html = app(SpatieLaravelMarkdownMarkdownRenderer::class)
    ->commonmarkOptions($arryWithOptions)
    ->highlightTheme('github-dark')
    ->toHtml($markdown);

To know more about these options, head over to the docs of Laravel Markdown.

In closing

We're very impressed with the quality of highlighting Shiki offers. Even modern PHP and JS code snippets are highlighted perfectly.

As mentioned above, Shiki can be resource-intensive. It's not recommended to run Shiki for every request (unless you like slow requests). If you're running Shiki locally, make sure you're using some form of caching.

Alternatively, you could opt to use a service like Torchlight. They take care of the syntax highlighting at their end, so you don't need to worry about any performance issues. They even got a commonmark package to get you started as well.

If you want to take care of syntax highlighting locally, check out our three packages:

• spatie/shiki-php
• spatie/commonmark-shiki-highlighter
• spatie/laravel-markdown

Our company website and this blog are open-sourced. Both sites make use of spatie/laravel-markdown Here's the code that converts Markdown to HTML:

• on our company website
• on this blog.

Be sure also to check out these Markdown related packages:

• spatie/sheets: stores and retrieves static content in plain text files
• spatie/yaml-front-matter: Can parse metadata at the top of a markdown file

I want to thank my colleague Rias, who kickstarted the work on the shiki-php package. Like always, Rias did an excellent job. You can see a few more examples of what Shiki can do, on his blog.

These packages aren't the first ones we've built. Check out this extensive list of packages our team has made previously. I'm pretty sure there will be something useful for your next PHP or Laravel project.