Tuesday, August 18, 2015

Dynamically load package contents with Dart's new Resource class

tldr: In Dart 1.12, you can now dynamically load the contents of files/assets from package dependencies. The new Resource class is currently implemented in the VM, with dart2js support coming in a future release.



Motivation

Dart applications are rarely just a collection of .dart files from a single developer. Real-world Dart apps often include numerous package dependencies, which contain additional Dart libraries as well as assets such as images, configuration files, template files, and more. It's always been possible to import third-party libraries with Dart's package: URI scheme, however prior to Dart 1.12 it was not possible to access non-Dart files via package: URIs.

For example, consider a tool such as stagehand, which generates new Dart projects such as web apps, server apps, pub packages, and more. The source of the new projects are stored as files and templates inside the stagehand package. Somehow, the stagehand tool needs to reference and load files from its packages. The contents of packages are addressable with package: URI scheme, but the Dart platform didn't provide a way to load those resources.

Hello, Resource!

Enter the new Resource class, which first appeared in Dart 1.12. The Resource class performs a very special role: allow Dart code to, at runtime, identify and load a resource identified by a package: URI.

Consider this simple package layout:

resourcetest/
  pubspec.yaml
  .packages
  bin/
    main.dart
  lib/
    myresource.txt

Here are the contents of main.dart:

main() async {
  var myResource = new Resource('package:resourcetest/myresource.txt');
  var resourceContents = await myResource.readAsString();
  print(resourceContents);
}

First, construct a new Resource with a package: URI. Of course, the Resource will only be able to load package: URIs from your application/script and dependencies declared in your pubspec.yaml file.

Then, call readAsString(), which returns a Future containing the contents of the resource.

That's it!

The Resource API

There are a few ways to load the contents of a Resource instance.

openRead() → Stream<List<int>>

Read the resource content as a stream of bytes.

readAsBytes() → Future<List<int>>

Read the resource content.

readAsString({Encoding encoding}) → Future<String>

Read the resource content as a string.

Technically, the Resource class can load any URI that it already knows how to load. For example, the Dart VM can load file: , http: , and package: URIs.

Things to consider

As of Dart 1.12, only the Dart VM has implemented the Resource class. Support is planned for dart2js in a near-future release.

The Resource class only works if your application is started with a valid way to resolve package: URIs. The Dart platform supports multiple ways to tell the runtimes and compilers where and how to resolve package: URIs, and under most circumstances, you don't need to do anything special.

One of the following must be true, in order for the Dart runtime to be able to resolve the package: URIs:


  • run your Dart app after it was installed with pub global activate
  • run your Dart app with a --package=path/to/.packages
  • run your Dart app with a --package-root=path/to/packages/dir
  • run your Dart app's entry point next to the .packages file
  • run your Dart app's entry point next to the packages/ directory


Normally, everything is set up for you, because pub does a great job of ensuring your app has a .packages file. This is true even with apps installed with pub global activate!

Summary

The new Resource class allows you to dynamically load the contents of package: URIs. You can use the new Resource class to load text or binary assets from within your app and your third-party dependencies.

Download a Dart 1.12-dev SDK and enjoy the new Resource class. We look forward to your feedback, and we can't wait to see what you build.

Tuesday, August 11, 2015

New Dart SDK helps eliminates symlinks

In which we retell the story of Dart and symlinks, investigate the new .packages file, and create a simpler world for Dart developers everywhere.

tldr: The Dart team is working towards a world where symlinks are no longer required because of the new .packages file.


Why a new solution for locating packages?

In the long long ago, during the before times (aka 2011), Dart could only run monolithic scripts and apps. In those early days, Dart didn't have support for packages/shared libraries. Then came package: URIs, which opened the door for sharing code. However, the platform only had one way to resolve that package: URI: look inside a packages/ directory next to the file importing the library. (Later, a --package-root option was added for more flexibility.) Thus began Dart's reliance on symlinks as a way of creating the packages/ directories next to all locations containing Dart scripts or libraries. Those symlinks and the package: scheme helped ushered in a boom of third-party libraries, the pub package manager, pub.dartlang.org, and more.

However, turns out that a reliance on symlinks leads to numerous issues, especially for our Windows developers. Putting symlinks into each subdirectory makes for a cluttered project. A directory full of symlinks only works on your local machine. The Windows operating system doesn't support symlinks (though, through some research, the pub authors did come up with a solution for modern Windows installs with something calls junction points). It became clear that the Dart platform needed a more robust and formal way to connect your script to your dependencies and third-party libraries.

The search began for a solution, and DEP5 (formally known as package spec) was born. The proposal specifies a new .packages file and a discovery workflow, and eliminates the need for symlinks everywhere. I am very happy to report that Dart SDK 1.12 will be the first version of Dart that doesn't require symlinks.

What is .packages?

The new .packages file is a simple map between a package name and a URI to the package's lib directory. Here is an example:

# This file has been generated by the Dart tool pub on Apr 14 09:14:14 2015.
# It contains a map from Dart package names to Dart package locations.
# Dart tools, including Dart VM and and Dart analyzer, rely on the content.
# AUTO GENERATED - DO NOT EDIT
unittest:/home/somebody/.pub/cache/unittest-0.9.9/lib/
async:/home/somebody/.pub/cache/async-1.1.0/lib/
quiver:/home/somebody/.pub/cache/quiver-1.2.1/lib/

The .packages file is auto-generated by pub, version 1.12-dev or greater.

Where does .packages live?

The .packages usually lives at the root of your package/app. For example:

my_app/
  README.md
  CHANGELOG.md
  .packages
  pubspec.yaml
  lib/
    my_lib.dart
  bin/
    my_app.dart

The Dart runtimes, such as the VM, dartjs, and analyzer, will traverse up directories, from the entry point script, to find a .packages. Thus, you do not need a .packages next to each script that you need to run or compile.

Do I check in .packages?

Do not check in .packages, because it contains paths that are local to your system. You can safely tell your source control system to ignore this file.

How do I use the file?

Under normal circumstances, you do not need to do anything special to use the file. pub creates the .packages file automatically.

The Analysis Server also uses .packages, so IDE/editor users (e.g. Atom, WebStorm, etc) should be able to work in a project without symlinks.

You can optionally specify a .packages as a command-line argument. For example:

dart --packages=path/to/a/.packages main.dart

Do I still need a packages/ directory in my project?

No. With .packages, there is no longer a need for symlinks or a packages/ directory in your project. Check out the next question for how to get to this beautiful new world.

Great! So, how can I eliminate symlinks?

You can use a hidden --no-package-symlinks command-line argument to instruct pub not to generate symlinks. For example:

pub get --no-package-symlinks

The above command will generate a .packages file in the root of your project, and will not generate any symlinks or packages/ directories.

Caveats

If you use WebStorm or IntelliJ, you may need to upgrade to WebStorm 11 EAP or the latest IntelliJ EAP to correctly work with .packages and no symlinks. The Dart plugin for JetBrains did add support for .packages, but it might not work with WebStorm 10. If you can't upgrade, do not run pub with --no-package-symlinks.

Next steps

Please install a 1.12-dev or later of the Dart SDK and test your code with the new .packages file. We eagerly await your feedback, especially from developers using Windows. You can also follow the meta-issue as we complete our implementation. Thanks!

Wednesday, July 8, 2015

Null-aware operators in Dart

Three new language features just landed in the latest dev channel build of the Dart! Collectively known as null-aware operators, these new features will help you reduce the code required to work with potentially null objects.

I'm excited for these new abilities, because typing less is always a good thing. Read on to learn more, and be sure to try these new features on Dart Pad.

??

Use ?? when you want to evaluate and return an expression IFF another expression resolves to null.

    exp ?? otherExp

is similar to

    ((x) => x == null ? otherExp : x)(exp)

??=

Use ??= when you want to assign a value to an object IFF that object is null. Otherwise, return the object.

    obj ??= value

is similar to

    ((x) => x == null ? obj = value : x)(obj)

?.

Use ?. when you want to call a method/getter on an object IFF that object is not null (otherwise, return null).

    obj?.method()

is similar to

    ((x) => x == null ? null : x.method())(obj)

You can chain ?. calls, for example:

    obj?.child?.child?.getter

If obj, or child1, or child2 are null, the entire expression returns null. Otherwise, getter is called and returned.

Try them today!

The easiest way to try null-aware operators is to use Dart Pad, our browser-based playground for Dart.

Or, download a 1.12-dev (or later) build of the Dart SDK to try these new features in the Dart VM, dart2js, and the Dart analyzer.

Saturday, April 11, 2015

Formatting Dart code before every git commit

Dart's dartfmt tool is a really neat utility to automatically format your code. Use the dartfmt tool in your workflow to ensure your code complies with the Dart style guide.

Of course, you don't want to manually run dartfmt. Instead, you want to automate it. Use git's pre-commit hook to ensure your code is formatted, before it is committed.

Add the following code to your .git/hooks/pre-commit script for your local repo, and make sure the script is executable.

#!/bin/bash

DARTFMT_OUTPUT=`dartfmt -w . | grep Formatted`

if [ -n "$DARTFMT_OUTPUT" ]; then
  echo $DARTFMT_OUTPUT
  echo "Re-attempt commit."
  exit 1
else
  echo "All Dart files formatted correctly. Yay!"
  exit 0

fi

If your code needs formatting, it will be formatted and written to disk. The commit will fail, so you have a chance to inspect the changes.

You can enforce formatting with your Continuous Integration system. Try these instructions to use dartfmt with Travis CI, and fail a build if a file isn't formatted.

Have you integrated dartfmt into your workflow? Let us know in the comments below!

Friday, May 9, 2014

I ported a JavaScript app to Dart. Here's what I learned.

In which I port a snazzy little JavaScript audio web app to Dart, discover a bug, and high-five type annotations. Here's what I learned.

[As it says in the header of this blog, I'm a seasoned Dart developer. However, I certainly don't write Dart every day (I wish!). Don't interpret this post as "Hi, I'm new to Dart". Instead, interpret this post as "I'm applying what I've been documenting."]


This post analyzes two versions of the same app, both the original (JavaScript) version and the Dart version. The original version is a proxy for any small JavaScript app, there's nothing particularly special about the original version, which is why it made for a good example.


This post discusses the differences between the two implementations: file organization, dependencies and modules, shims, classes, type annotations, event handling, calling multiple methods, asynchronous programming, animation, and interop with JavaScript libraries. Finally, I detail the lessons learned.

Background



Mr. Paul Lewis wrote a snazzy little web app for audio processing. His Music DNA app uses JavaScript to load up an audio file, play it, and visualize the sounds in a browser. It’s a cool little demo of what’s possible on the web platform.


Screen Shot 2014-04-25 at 4.10.53 PM.png


I’m always impressed with Paul’s designs and code, so I asked myself: Could I easily port this app to Dart? Was Dart up to the challenge of a web audio app? Is there benefit to using Dart for little apps like this?


Spoiler: yes, yes (mostly), and yes. Read on!

Fine print



I ported the original JavaScript version of Music DNA app as of this revision. The code for the  Dart version is also available. It’s possible that by the time you read this, the original JavaScript version will have changed. I also attempted to minimize the differences between the two code bases. I tried to keep the names the same (when appropriate), and the number of files the same (again, when appropriate).

Organization



In which I examine file layout and navigation of both the original project and my Dart version.

Original version



The original version collected all files into a single directory.


Screen Shot 2014-04-25 at 9.48.35 PM.png
Original version


To discover the “main” JavaScript file, I opened index.html and I found six script files:


Screen Shot 2014-04-25 at 9.49.50 PM.png
Original version


Hm, where does the application start? I wasn’t sure. I opened each file, in order. It appeared that bootstrap.js contained the “entry point” of the application.


The entry point was a “immediately invoked function expression” (IIFE):


music-dna_bootstrap_js_at_master_·_paullewis_music-dna.png
Original version

Dart version



Dart’s package manager, pub, defines a layout convention for packages and apps. The Dart version has the HTML, and Dart files for the app, inside a web directory. The library files (files that contain libraries that might be useful for any app) are inside a lib directory. (More on this soon.)


Screen Shot 2014-04-25 at 10.34.11 PM.png
Dart version


To find the “entry point”, I opened the main HTML file (music_dna.html). I found a single Dart file and two supporting JavaScript files.


Screen Shot 2014-04-25 at 9.59.01 PM.png
Dart version


I did not port the id3.js file to Dart for two reasons: it was minified, and I wanted to test Dart-JavaScript interop. The dart.js file is boilerplate: it helps the app run in both JavaScript engines and Dart VM.


Dart_Editor.png
Dart version


The application starts from the main() function in bootstrap.dart. All Dart programs start at the main() function.

Dependencies and modules



In which I examine how both versions handle linking to libraries or modules.

Original version



JavaScript doesn’t have a native library or module system. The original version listed each “module” as a separate script tag from the main HTML file (screenshot above). The order of the scripts is important, as scripts that load later depend on scripts that were loaded earlier. The code does not explicitly declare its dependencies.


For example, the bootstrap script creates a new instance of MusicDNA. The bootstrap.js file does not include a reference to the location of the definition of the MusicDNA function.


music-dna_bootstrap_js_at_master_·_paullewis_music-dna.png
Original version


The MusicDNA function comes from the music-dna.js file, which is loaded before the bootstrap.js file via script tags.

Dart version



The Dart language natively supports libraries. A library can import other libraries. For example, the bootstrap.dart file imports its dependencies:


Screen Shot 2014-04-25 at 10.31.14 PM.png
Dart version


The Dart SDK has some standard libraries like dart:html for DOM access, dart:js for interop with JavaScript (more on this below), and more. The code also imports music_dna.dart which explicitly provides the MusicDNA class. Using the show keyword with import makes it easy to glance at what names are provided by what library.


The music_dna.dart file is its own library, which also declares its own dependencies:


Screen Shot 2014-04-25 at 11.10.20 PM.png
Dart version


Notice the two types of URIs used in imports: dart:____ and package:_____. Imports from dart: come from the Dart SDK, and imports from package: refer to libraries that live in packages. A package is a bundle of Dart code, conforming to some layout conventions, and declared by a pubspec.yaml file.


Both libraries and applications can live inside of a package. The music_dna application is itself a package. The music_dna library imports the audio library from its own package. This is a handy way to avoid using relative (and brittle) paths that would have to reach up and out of the file’s directory.


Back to libraries: A library can be split into multiple parts. The audio.dart file declares the library, its imports, and the two other files that contain the library’s classes: audio_parser.dart and audio_renderer.dart.


Dart_Editor.png
Dart version


An abbreviated way to look at the libraries and dependencies of the Dart version:


Dart version

Shims



In which I explore how both versions deal will shimming browser APIs, such as Web Audio and requestAnimationFrame. A shim is a thin layer of code that compensates for browser differences in function names or APIs.

Original version



The original version shimmed the APIs using a common JavaScript language pattern. For example, here is the shim for AudioContext (from Web Audio) in audio_parser.js:


music-dna_audio-parser_js_at_master_·_paullewis_music-dna.png
Original version


The original code also shimmed requestAnimationFrame in bootstrap.js:


Screen Shot 2014-04-25 at 10.44.26 PM.png
Original version

Dart version



The dart:html library hides the shims, so I don’t need to worry about it. For example, I just needed to import the dart:web_audio library and then use the AudioContext name.


Dart_Editor.png
Dart version


Dart_Editor.png
Dart version


When I compile the Dart app to JavaScript, the shims are included in the generated JavaScript. This ensures the generated code works in all modern browsers.

Classes



In which I examine how both versions model and construct concepts.

Original version



JavaScript is not a class-based object oriented language. The original version used standard JavaScript patterns for “classes”.


music-dna_audio-parser_js_at_master_·_paullewis_music-dna.png
Original version


The JavaScript version initializes fields by constructing new objects as well calling methods. The object isn’t initialized until the entire body of the AudioParser function is run.

Dart version



Dart is a class-based object oriented language. I created a formal class for AudioParser.


Dart version


The Dart version initializes most fields inside of the constructor body. By the time the constructor body is run, the object is initialized.

Type annotations (aka discovering a bug)



In which I use type annotations and discover a bug in the original code.

Original version



JavaScript doesn’t have a way to explicitly state that a variable can hold a specific type of object. Fields are simply marked as var, as in this example:


Screen Shot 2014-04-26 at 12.26.34 AM.png
Original version


It’s unclear what kind of function can be assigned to audioDecodedCallback. I can assume what sourceNode or audioRenderer can be, because I know this is a Web Audio app and there’s another file called audio-renderer.js.


Method parameters also do not have a type annotation. Here is one example:


Original version


Can you spot the bug in the above example? I didn’t, until I converted the code to Dart.

Dart version



Type annotations are optional in Dart, but I love ‘em. I use type annotations as inline and explicit docs for my fellow programmers and my productive tools.


We’ve seen it before, but here’s the recap of how I use type annotations for fields in AudioParser:


Screen Shot 2014-04-26 at 12.36.44 AM.png
Dart version


So, gainNode is of type GainNode. I could probably have guessed that, if I knew there was a Web Audio type called GainNode. Without the type annotation, editors and analyzers can’t infer that gainNode is a GainNode. And that’s important, because…


Dart version


Dart Editor and the Dart analyzer knows that disconnect() takes an int and not a GainNode. A warning is displayed, and bug identified!


Turns out, the code didn’t need the call to disconnect() anyway. But, I feel more confident knowing that tools can identify potential errors.

Event handlers, and calling multiple methods on the same object



In which I encounter a Dart-y DOM, use a Dart language feature, and reduce the amount of code I have to type.

Original version



The traditional way to listen for DOM events (like drop or dragover) is to use addEventListener(). This method takes a string as the first parameter. If you misspell the name of the event, you will not get a warning, and the program will keep on trucking. There is no code completion for the values of the strings, because the language can’t express all the valid values for the first argument.


Here is an example, from the original code, of listening to multiple events on a single fileDropArea object.


Screen Shot 2014-04-26 at 12.47.25 AM.png
Original version

Dart version



The dart:html library offers DOM APIs that are more toolable. There are specific methods for specific events, for example onDrop. If you misspell the method name, the tools can warn you about an unknown method. Also, you can use code completion to help discover other events to listen for.


Dart supports method cascades, which help reduce repetitive code. If you need to call multiple methods on a single object, cascade the calls and avoid repeating the variable.


Here is the Dart code for setting up the drag and drop area:


Screen Shot 2014-04-26 at 1.12.45 AM.png
Dart version


Notice how fileDropArea is stated only once. The double dot (..) syntax is the cascade. Compare this version to the original JavaScript version above.

Asynchronous callbacks



In which I explore how both versions handle asynchronicity.

Original version



When something needs to run “later”, JavaScript calls a callback function. The MusicDNA app uses asynchronous callbacks when decoding audio data.


  1. MusicDNA has a onAudioDataParsed function.
  2. MusicDNA creates a new AudioParser, and passes onAudioDataParsed to it.
  3. MusicDNA calls audioParser.parseArrayBuffer(), which calls an asynchronous decodeAudioData method.


Control is returned to the browser, and events are processed as normal. The audio is decoded in the background. Sometime later...


  1. When decodeAudioData is finished and is successful, it calls audioParser.onDecodeData.
  2. When onDecodeData is finished, it calls onAudioDataParsed.
  3. Now we’re back in MusicDNA!


As you can see, a function can get passed around, with no clear line of execution.


Here’s the code, see if you can follow the callback:
Original version


In AudioParser:
Original version


parseArrayBuffer() calls decodeAudioData() which asynchronously calls onDecodeData.
Original version


The onDecodeData function calls the callback given to AudioParser from the constructor:


Original version

Dart version



Dart’s core SDK has a Future class, which represents values available sometime in the future. An asynchronous function or method returns a Future instead of accepting a callback as a parameter. The Future completes when the asynchronous function is finished, and a value is available.


In the Dart version, parseArrayBuffer returns a Future, as seen below:
Dart version


Now that the async methods can return futures, the code can get simplified. For example, Futures are chainable, so music_dna.dart looks like this:


Dart version


The above code reads:


  1. Read the file into an array buffer.
  2. When the loadEnd event is done, then parse the array buffer.
  3. When the array buffer is done, then read the duration.
  4. Return a future to caller of parse knows when parsing is finished.

Animation



Modern web apps animate with requestAnimationFrame, which is an API that lets your app get notified when the browser is ready for your app to draw a frame.

Original version



Again, because JavaScript uses callbacks for async notifications, the requestAnimFrame (a shimmed version of requestAnimationFrame) uses callbacks.


Original version

Dart version



Dart uses Futures in many places, including in the HTML libraries. Specifically, requestAnimationFrame has been rewritten to return a Future that completes when the frame is ready.


Dart version

Code size


There are two ways to look at code size: the original code that the developer works with, and the compiled, minified, gzipped code size. It's important to stress that neither the original version or the Dart version made any attempts to create the most terse code possible. Line counts include blank lines and comments. I believe this app is too small to make any reasonable claim about lines of code correlation to language.

Original version


The important JavaScript files, their sizes, and number of lines:
  • audio-parser.js: 1823 bytes, 78 lines
  • audio-renderer.js: 2751 bytes, 120 lines
  • bootstrap.js: 1803 bytes, 69 lines
  • music-dna.js: 3608 bytes, 130 lines
Total bytes for the scripts (not counting id3.js): 9985 bytes.

Total lines for the scripts: 397 lines.

The original app did not concat the JS files or minify them, however most developers should perform those steps with "real" apps. Most web servers will gzip on the fly, so it's worth noting that the total number of gzipped bytes from scripts (gzipping each file individually) results in 3825 bytes.

Dart version


The important Dart files, their sizes, and number of lines:
  • lib/audio.dart: 427 bytes, 12 lines
  • lib/src/audio_parser.dart: 1786 bytes, 68 lines
  • lib/src/audio_renderer.dart: 2273 bytes, 99 lines
  • web/bootstrap.dart: 1763 bytes, 71 lines
  • web/music_dna.dart: 1623 bytes, 59 lines
Total bytes for the scripts (not counting id3.js): 7872 bytes.

Total lines for the scripts: 309 lines.

Compiling and minifying the Dart scripts to JavaScript generates 143kb bytes. Gzipping the generated JavaScript (as most web servers and browsers compress/decompress on the fly) results in 41kb bytes. (For comparison, jQuery itself is 32kb minified and gzipped. However, the original Music DNA app does not use jQuery.)

Interop with JavaScript libraries



The Music DNA app uses a JavaScript file called id3.js to read the ID3 tags and extract metadata like song title and artist.

Original version



The id3.js file is included as a script tag, just like any other JavaScript file.


Original version


Using the ID3 functionality is easy, as it’s just JavaScript in JavaScript. Here is how the ID3 library is used inside of bootstrap.js:


Original version

Dart version



Using JavaScript files in Dart requires the dart:js library, which provides Dart-JavaScript interop. I had to import the dart:js library, explicitly get a reference to the ID3 object (which originally came from the id3.js file), and use the callMethod method to call JavaScript methods.


I was able to use id3.js in the Dart app, but the integration isn’t as trivial as it is with JavaScript-JavaScript interop.


Here is the Dart code:


Dart version

Lessons learned



Porting the Music DNA app from JavaScript to Dart was a lot of fun. Here’s what I learned:


  • Futures are great! They really simplify API design.
  • Type annotations save the day. I found a bug in the original code after I ported it to Dart, thanks to type annotations and code analysis.
  • You can interoperate with JavaScript from Dart, it’s just not a “cut and paste” job.
  • It’s easy to tell where Dart programs start.
  • Compiled, minified, and gzipped apps about this size end up a little bigger than jQuery.
  • Dart’s library mechanism makes it easy to determine where names come from.
  • Dart’s HTML library shims APIs like requestAnimationFrame for you.
  • Method cascades are sweet.
  • Code completion. Love it. Especially helpful when exploring new APIs.

There are other language and library features of Dart that I didn't get a chance to use. If you're interested, there is a Dart language tour that might be interesting to you.

Of course, most of the techniques used by Dart in this article aren't impossible with JavaScript. In fact, future versions of JavaScript should get features like modules and promises. They just aren't built-in, obvious, or easy "out of the box" with JavaScript as we know it today. And this is where I begin to see some of the real power with Dart: the out of the box experience is really good. Shims? yup. Futures? yup. Layout convention? yup. Classes? yup. Type annotations (instead of comment)? yup. Dart seems to have a broader and more functional base on which we can build. I like it.


Many thanks to Paul Lewis for the inspiration and sharing his code.

Disclaimer

I'm probably required to say that the views expressed in this blog are my own, and do not necessarily reflect those of my employer. Also, except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the BSD License.