Daily Archives: May 2, 2017

The core of a Docker image is basically a compressed set of files, a set of .tar.gz that (if you look in your Docker image folder on your host machine, you will see. We are going to use this local repository for this first set of methods.

The Docker engine communicates with the Docker Hub via the Docker Remote API, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. This first method does not require having Docker installed on your machine. Lets say that I want to bootstrap tensorflow from Docker. First I should create the tensorflow image:

Note that if you want (much) more detailed output for debugging to the console, you need to enable --verbose mode:

Now I can shell into it, and import tensorflow:

Docker has two commands in the DOCKERFILE that have something to do with execution, CMD and ENTRYPOINT. The differences are subtle, but the best description Ive found is the following:

A CMD is to provide defaults for an executing container.

and

An ENTRYPOINT helps you to configure a container that you can run as an executable.

Given the definition, the ENTRYPOINT is most appropriate for the Singularity %runscript, and so using the default bootstrap (whether from a docker:// endpoint or a Singularity spec file) will set the ENTRYPOINT variable as the runscript. You can change this behavior by specifying IncludeCmd: yes in the Spec file (see below). If you provide any sort of %runscript in your Spec file, this overrides anything provided in Docker. In summary, the order of operations is as follows:

Do a barrel role! Use a spec file! Many times, you want to bootstrap an image, and then either change the %runscript or add additional software or commands in the %post section. To achieve this, you can create a specification file. Currently, these are distributed with the naming format [myfile].def, however (soon) we will use a standard name, Singularity so all specification files can be automatically found. Here is what the spec file would look like for tensorflow:

In the example above, I am overriding any Dockerfile ENTRYPOINT because I have defined a %runscript. If I want the Dockerfile ENTRYPOINT to take preference, I would remove the %runscript section:

Note that the spec file above would be (almost) equivalent to the command:

minus the useless echo at the end. If I want the CMD to take preference, I would add IncludeCmd:

The solutions above would be ideal for saving a custom specification of an image to build at some runtime.

For both import and bootstrap using a build spec file, by default we use the Docker Registry index.docker.io. Singularity first tries the call without a token, and then asks for one with pull permissions if the request is defined. However, it may be the case that you want to provide a custom token for a private registry. You have two options. You can either provide a Username and Password in the build specification file (if stored locally and there is no need to share), or (in the case of doing an import or needing to secure the credentials) you can export these variables to environmental variables. We provide instructions for each of these cases:

You can simply specify your additional authentication parameters in the header with the labels Username and Password:

Again, this can be in addition to specification of a custom registry with the Registry parameter.

You can export your registry, username, and password for Singularity as follows:

If you are having trouble, you can test your token by obtaining it on the command line and putting it into an environmental variable, CREDENTIAL:

This should place the token in the environmental variable TOKEN. To test that your token is valid, you can do the following

The above call should return the tags list as expected.

Finally, we can achieve a shell experience, meaning shelling into Docker image imported into Singularity. We do this by storing the entire image in a temporary location, and then running the same function. You would do something like this:

A common use case is to want to start with a Docker image, possibly add custom commands, and have a Singularity image when you finish. You can read a bit about bootstrapping here to get a sense of adding the custom commands and software. To specify Docker as the build source, you simply need this header:

In the case of omitting the tag (latest) it is assumed that you want the latest image. In the case of omitting the namespace (library) it is assumed that you want the common namespace, library. If you have a reason to use the Docker Engine, we also have a method to do this. The benefit of this method would be that you could use an image built locally (in your local cache) that isnt on Docker Hub.

Here we will access Docker images via the Docker command line tool, meaning using the Docker engine. As is the Docker standard, the image is first looked for in your local cache, and if not found, is pulled from Docker Hub.

We wrapped this entire process into a Docker container itself, which means that you can use a Docker container in a Docker container to export a Docker container into Singularity! Nuts. Full instructions are provided, however here is the gist:

How did this come to be? It so happens that Docker has an export command to pipe this data out, and Singularity has an import command to take them in. Thus, you can do a simple import of a Docker image into a Singularity command by doing:

Where $container_id is the id of a running container obtained with docker ps. However, there are subtle details like the environment and permissions that this method will miss. Its also the case that most Docker images dont run (and stay running) easily unless you do something like:

Early on we created a docker2singularity.sh, a script that you can download and run as follows:

To produce a Singularity image of ubuntu:latest in the present working directory.

Why wont my image bootstrap work? If you cant find an answer on this site, please ping us an issue. If youve found an answer and youd like to see it on the site for others to benefit from, then post to us here.

This entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo.

Excerpt from:

Singularity and Docker | Singularity

The struggle for electric vehicles to gain legitimacy in a world dominated by supersized SUVs and overbearing big rigs is something of a David and Goliath story.

The underdog just landed a blow right between the eyes of Americas major car manufacturers.

In April, Tesla became the most valuable automaker in the US, passing General Motors in total market value. GM has since regained the lead by a slim margin, edging Tesla in market cap by a little more than a billion dollars.

Still, the ability of a 15-year-old company to stand shoulder-to-shoulder with the venerable Detroit automakers is extraordinary. Its a reflection of the broader picture of how alternative energy and renewables have risen to prominence. It would seem to be no coincidence that at a time when electric vehicles appear poised for widespread adoption, solar and renewable energy have become cheaper than coal.

Tesla is set to roll out its most affordable EV to date later this yearthe Model 3, retailing at $35,000 before tax breaks. The Model 3 reportedly has more than 400,000 pre-sales. There are only about 540,000 EVs on US roads today, according to report called The State of EV Charging in 2016, produced by electric car charging companyChargePoint.

The release of the Model 3 alone promises to nearly double the number of electric vehicles (including hybrids). Thats not quite as impressive as it first sounds considering there were more than 260 million vehicles in the US as of 2014, according to the Bureau of Transportation Statistics. In other words, EVs will still account for less than one percent of all vehicles.

There is potential for EVs to reach a market share of 30 percent or greater by 2030, but that will require radical shifts in environmental and regulatory policy, according to John Axsen, an associate professor at Simon Fraser University in British Columbia who studies green technology, consumer behavior and environmental policy.

As long as gasoline vehicles are able to belch pollutants and greenhouse gas emissions for free, then EVs will have a disadvantage, he says by email to Singularity Hub. Economists call this a market failure. So nearly all the good research out there shows that strong policy is responsible for any success we've seen so far, and that we'll need more strong policy to see any real success going forward.

There are very few places where such policies exist. For example, Norway has reached 25 percent market share for EVs because it has huge taxes on fossil fuels, huge taxes on conventional vehicles, and very substantial financial and non-financial incentives for EVs, according to Axsen.

California is leading the way in North America. Its zero-emissions vehicle (ZEV) mandate requires automakers to sell a certain percentage of electric vehicles. Quebec has recently followed suit with its own ZEV mandate. I believe that if California did not have this policy, Tesla would never have existed (and Toyota probably never would have developed the Prius, for that matter), Axsen says.

Earlier this year, Californias utilities submitted plans to collect up to $1 billion from customers to expand the states EV infrastructure. If approved, California would add more than 10,000 new charging stations, as the state pushes to put a million EVs on the road by the end of the decade. The Guardian reported last year that the European Union has created a draft directive requiring every new and refurbished home in Europe to be equipped with a recharging point.

Another impediment to EVs reaching a tipping point is supply, according to Axsen. There are relatively few makes and models available, particularly in truck and van classes. And then, many car dealerships are not carrying these EVs in their inventory, and research shows that many dealers in Canada and the US are not even trying to sell the EVs to customers, he notes.

Most of the major US automakers manufacture EVs or hybrids, with more than 55 models on the market. There are also a number of emerging startups vying to become the next Tesla. Some of the best-funded are in China, while a company called Lucid Motors out of California (of course) has been touted as a potential challenger to Tesla. Its luxury model, still not in full production, boasts 1,000 horsepower and can go 400 miles on a single charge.

Elon Musk and Tesla, meanwhile, are not satisfied with just building the worlds most advanced EV compact cars and sedans. The company recently announced it would next tackle a mini-bus, pickup truck and even semi-truck. The latter would be a particularly disruptive technology, especially if Tesla outfits it with the companys Autopilot system.

Of course, there are quite a few technological roadblocks the company will need to address, not least of which is developing a battery system that can handle a heavy, long-haul 18-wheeler rig. A company not named Tesla but called the Nikola Motor Company thinks it has the answer by using hydrogen fuel cells to power a fully electric 18-wheeler.

It claims its Nikola One will have a range of 800 to 1,200 miles while delivering 1,000 horsepower. And just a few days ago, Toyota also announced plans for a hydrogen cell-powered big rig, in a new race to produce the first zero-emission 18-wheeler.

Electric vehicle technology is getting better, and a few regions are showing the market potential, Axsen says, such as California, Norway and the Netherlands. Both European countries plan to phase out fossil fuel vehicles by 2025.

Despite such successes, Axsen emphasizes that without changes in policy, the EV market will likely continue to hover around one percent, perhaps hitting 10 percent by 2030. The strongest policies, which encourage automakers to sell a wide variety of EVs, can push market share as high as 30 to 45 percent by 2030, he adds.

The current US administrations efforts to rollback Obama-era fuel economy rules would seem to imply that policies favoring EVs will remain status quo at best for now. That could mean China and Europe will speed past the US and Canada in the widespread adoption of electric vehicles.

While Axsen doesnt share Musks unfettered optimismpossibly because he is not beholden to stockholdershe does think the drive to a zero-emission transportation system is navigable in the US and Canada.

Though our current policies are not nearly up to the task, governments have proven options at their disposal that will get us where we need to go, he wrote previously.

Image Credit: Shutterstock

Read more here:

Electric Vehicles Are on the Rise: Here's How to Sustain Their Growth - Singularity Hub