MyGet Build Services - Package Versioning Explained

At MyGet, we do everything possible to reduce friction. It's in our DNA! Package versioning is currently the biggest roadblock you'll encounter when setting up a solution to automate package creation.

Versioning semantics

A NuGet package is uniquely identified by its package identifier and its version. This implicit requirement is often overlooked upfront and requires some thinking to define a proper versioning strategy. Luckily for us, some smart people already came up with something called Semantic Versioning (SemVer). We strongly believe in SemVer's pragmatic approach towards API versioning and the benefits that come with it.

Even though NuGet does not fully support the entire specification yet (the SemVer specification is still RC), it already has some features that allow you to benefit from using SemVer. An example of that is the Update-Package -Safe command option, which allows you to update your consumed packages to the latest safe package version: updates are constrained to the same Major and Minor package version. The only drawback of this feature is that you have to count on a safe versioning scheme applied by the package author. This is not guaranteed on the NuGet Gallery.

Build Source Configuration

If you set up MyGet Build Services for your feed, you'll get automatically created packages pushed to your feed. This implies automatic versioning. We've added some new capabilities to our Build Services in our recent MyGet v1.6 release. For one, we've added a UI to define the versioning scheme to be used.

You are in control

The biggest improvement is the fact that the build source now remembers the build counter for you, so you don't have to deal with maintaining a version.txt file or anything similar in your source repository. Of course, if you really want to, you still can: we're removing friction, not features! The build counter can easily be reset using the reset link. Similar to what popular build servers do (and any decent build server should do), you'll be able to define a versioning scheme using a placeholder for the build counter. You've now gained more control over the versioning "magic" we applied before.

In addition, you can gain full control by hooking into the build process. MyGet Build Services scans for the following source artifacts (in order of precedence):

If you provide your own build.bat script or MyGet.sln, you specifically instruct MyGet Build Services how to act on your sources. This also means you'll need to take care of versioning yourself. That's why we provide you with the following set of parameters so you can benefit from using the versioning scheme you already defined, as well as the build-counter attached to your build source. Note that these environment variables are read-only and are reset to the initial values at the start of the build process.

%BuildRunner% Always "MyGet", can be used to determine if running on MyGet Build Services
%NuGet% Path to a maintained-by-MyGet NuGet.exe
%SourcesPath% Path to source code being built
%Configuration% Build configuration (defaults to Debug)
%Platform% Platform to build (defaults to blank)
%VersionFormat% Version format specified in build configuration
%BuildCounter% Build counter value
%PackageVersion% %VersionFormat% with %BuildCounter% filled in, used as the auto-generated package version number
%EnableNuGetPackageRestore% NuGet package restore enabled? Always true.

 

Package Source Configuration

We've also pulled Package Sources out of beta for everyone to enjoy, including our Free users! Package sources play a key role in the MyGet Build Services workflow. By default, the NuGet Gallery is configured as an upstream package source. You could obviously also configure Chocolatey, an Orchard feed, or your own MyGet feed, or all of them! These upstream package sources allow you to reference or mirror packages onto your own feed. It also allows you to easily push packages from your feed to the the upstream package source, given you configured your API key in the package source configuration.

When you want to push a package upstream, you'll be prompted with a dialog asking you which upstream feed should be used. In addition, you'll also get the opportunity to modify the package version as it should appear on the upstream package source. Note that no compilation is happening, we're simply adjusting the package version if you changed it. Because semantically nothing changed to the package, you'll only be able to change the -PreRelease tag, or simply remove it and make your upstream package a full release.

Without enforcing them

We won't enforce you in using SemVer as this would introduce friction for those who don't want to use it. It's not up to the repository host to define your versioning approach: this is the responsibility of the feed owner. This is impossible on NuGet.org as the feed is owned by the Community, which is definitely not organized around a single common versioning scheme.

However, if you - as a MyGet feed owner - want to enforce it, then you can by simply checking the appropriate checkbox in your feed settings. We try to find a balance between removing friction and providing freedom in your choices, so we hope you like it! Don't hesitate to send us your feedback or log them on our UserVoice.

Happy Packaging!

Author: Maarten Balliauw on 06 Mar 2013

Package sources feature out of beta

Good news: our package sources feature is out of beta after almost a year. The reason for that is that we wanted to make this a very stable feature which it proved to be thanks to many of our users testing it. But… what is this feature?

By default, MyGet uses the official NuGet gallery as the one and only source for packages. Package sources allows us to add additional sources for packages. For example if we want to use Chocolatey as a source for packages, we can add it as a primary or secondary package source. What’s more, we can also push packages from our MyGet feed to the upstream package source.

Package sources list

Let’s dive into some of the capabilities of Package Sources.

Adding a package source

Under the Package Sources tab of our feed, we can click the Add button to create a new package source. We can specify a name and URL to the package source, optionally provide authentication details and an API key for the package source. We can also select a preset package source, for example Chocolatey, a TeamCity server or a list of Windows 8 packages available from NuGet.org.

Add package source

Optionally, we can also provide a filter. Filtering is based on the OData Filtering System. Valid filters are similar to Id eq 'jQuery' or IsLatestVersion eq true and Id ne 'Foo'. If we wanted to only be able to add packages which have the term “javascript” in their description, we can add the following filter: substringof('javascript', Description) eq true.

Throughout the context of your feed, MyGet will use the package sources in an ordered fashion. Reordering package sources to have a different default, for example, can be done by simply dragging items in the list.

Adding packages from another feed

After adding a package source, we can use it when adding packages to our own feed on MyGet.

Add package from another feed

We can select the package source to use from the dropdown and search the selected package source.

Feed proxying

Ever thought about having your own MyGet feed containing your company’s internal libraries and combining them with perhaps a filtered subset of the official NuGet package source? MyGet provides such feature out of the box, built on package sources.

Proxy an upstream package source

When creating or editing a package source, we can tick the “Proxy package source” checkbox:

Proxy feed

Note that also with the feed proxy, filtering the upstream feed is possible. For example, the filter string substringof('wp8', Tags) eq true that we used will filter all upstream packages and only include those where the tags contain “wp8”.

From this point forward when searching packages in Visual Studio, we’ll be able to see our own packages enriched with packages from the official NuGet package source:

Add package from feed

Instead of working with a number of NuGet feeds, your development team will just work with one feed that is aggregating packages from both MyGet and other package sources out there (NuGet, Orchard Gallery, Chocolatey, …). This centralizes managing external packages and makes it easier for your team members to find the packages they can use in your projects. Management of packages that can be used by your team is centralized by this feature.

Push upstream

Each time a team member of your open source or enterprise project commits source code changes, your build server pushes an updated release to this package repository in the form of a prerelease NuGet package. Now what happens if a release to the official NuGet package source has to be created? Typically, you will either create a fresh package which will be the package to release, or download a package from your build server, change the version and upload that one to NuGet.org (or another repository). No need for such bloated process: MyGet will perform the push for you.

The first time you want to push a package to another NuGet feed, you’ll probably have to configure the other feed’s URL and API key to use when pushing there. In the package source, simply make sure to provide an API key and you’re all set to make use of this feature. From the moment a package source has been configured, using the “Push” button will enable you to push packages to another feed.

Push package upstream

After clicking the “Push” button, MyGet will present you with an overview of the package which will be pushed to another feed. Select the feed to which you want to push and verify the other fields. You can also modify the prerelease tag if needed (for example for promoting a prerelease to a stable package version).

Push package from MyGet to NuGet

Click “Push” and MyGet will take care of pushing the package to the selected package source.

Happy packaging!

Author: Maarten Balliauw on 04 Mar 2013

New features in MyGet Enterprise

The MyGet Enterprise plan was introduced a couple of months ago and has had several management options since then. After logging in to your MyGet Enterprise instance the main menu displays a Dashboard link which gives the plan administrator access to a management dashboard.

MyGet Enterprise administration

In the management dashboard, MyGet Enterprise administrators can view statistics on their instance (number of feeds, storage usage, some aggregates on consumption and so on) as well as manage several options for their instance such as managing users, feeds, Google Analytics integration, SymbolSource integration, SMTP settings and quota management. With the 1.6 release of MyGet, we’ve added several other options to this enterprise management dashboard. And of course, your MyGet Enterprise plan also contains all of the enhancements that were introduced on the public MyGet website as well (such as retention policies, feed activity logs, build services, download as ZIP and much more).

Block e-mail addresses not belonging to the organization

Enterprise users not using Active Directory integration but instead relying on social identity providers like Microsoft Account or Google Account previously had no option for blocking users not belonging to their organization from creating an account on their MyGet Enterprise instance. Starting today, domain names for e-mail addresses can be whitelisted or blacklisted.

image

If your enterprise has Google Account e-mail addresses which all end in “@acmecorp.com", adding a whitelist entry for “acmecorp.com” will allow only those users to create an account on your MyGet Enterprise plan. Multiple entries can be entered, separated by a semicolon.

Users can be made administrator

Before our 1.6 release, multiple administrators per MyGet Enterprise instance were already a possibility but could only be managed by sending our support people an e-mail. Because this is a common task, we now made it possible for existing MyGet Enterprise administrators to grant or revoke administration privileges to users in their organization.

image

User removal

What happens if a user for a MyGet Enterprise instance leaves the organization? With our new release, users can be deleted from a MyGet Enterprise instance by clicking the Delete button on the user management pages.

image

By default, the user and its feeds will be removed. This is not always something you will want to do: what if that user is the owner of a production MyGet feed? The delete user dialog allows you to transfer feed ownership for all of the user’s feed to another user in your organization.

More information about our MyGet Enterprise plan can be found on the MyGet Enterprise plan page.

Happy packaging!

Author: Maarten Balliauw on 01 Mar 2013

Support for feed names with dashes and underscores

An often requested feature for MyGet was support for feed names with dashes and/or underscores. With our 1.6 release we introduce just this: support for feed names with – and _ in their name.

Feed names with dashes and underscores

Many people asked us for this feature. The main scenario these users referred to was having feeds for development and production, having feeds per branch and so on with a recognizable prefix.

Since we deployed it in our staging environment, we’ve been using it continuously ourselves. Every support request that comes in and requires investigation is prefixed with “support-“. When we develop a new feature and test it on a given feed, we typically prefix those feeds with “dev-". I even saw a “chuck-norris-was-here” feed appearing during development. We hope you like this as much as we do!

Happy packaging!

Author: Maarten Balliauw on 27 Feb 2013

Release notes for MyGet 1.6

Many users have asked for us to provide release notes for everything we put online. Since that’s a great idea we’ll start doing this with the current 1.6 release which has been deployed earlier this week. But before we dive into that, let’s give you some hints about our development process and versioning system.

Our development process and versioning system

If you look at the footer of our website, you’ll have seen the following for the past few weeks already:

image

That’s right: we’ve been on (parts of) version 1.6.x for the past few weeks already. The reason for that is we version after our sprints and do continuous deployment of most things we work on. This means that when we start deploying features for our v1.7.x sprint, we’ll already update the version number in the footer to that number yet we’ll provide release notes only at the end of our sprint. So while today, you may already see v1.7.x mentioned in our footer because we've already deployed some features from that sprint, the following release notes are valid at time of writing this post.

Release notes for MyGet 1.6

MyGet 1.6 was released on February 25, 2013.

Features

MyGet

  • Minimum length for usernames has been decreased to 3 characters (previously 6). Shorter usernames are now possible.
  • A new menu item under feed: "Feed Settings" will contain settings specific to how MyGet handles packages for a given feed.
  • Feed settings contains an option to enable/disable overwriting of packages on the feed
  • Feed settings contains an option to enable/disable validation on the package version number with regards to Semantic Versioning
  • Dashes and underscores in feed names are supported. Feeds can be named foo-prod for example.
  • Download feed as ZIP.
  • Package Sources are out of beta.
  • API key in package source configuration is now masked

MyGet Enterprise

  • Block e-mail addresses not belonging to the organization.
  • Users can be made administrator.
  • User removal (with the option of transfering their resources to another user).

MyGet Build Services

  • Copy build log to clipboard.
  • Specify build configuration and platform (Release/Debug and Any CPU/Mixed Platforms/...)
  • Support incremental build numbers.
  • Support configuring the build number using a template. Register version number as an environment variable.
  • Support building Windows Phone 8 projects.
  • Hotlink commit on GitHub/BitBucket from the build list.
  • Refresh build status automatically.
  • Support creating tools packages.
  • Hanging build detection.
  • Install psake on the build servers.

Bug Fixes

  • Copy to clipboard on feed details page did not work in Chrome version 24.0.1312.57 and up.
  • Feed statistics are not updated in some situations.
  • NuGet Package Explorer always shows prerelease packages in the feed list.
  • Build Services: building from protected SVN repositories isn't always working.
  • Build Services: GitHub API only returns 30 repositories.

Happy packaging!

Author: Maarten Balliauw on 25 Feb 2013

We were down...

On 23 February 2013, we’ve been down for 8 hours. We received a monitoring alert (and some e-mails and tweets as well) at around 9:50 PM (CET) notifying us of this event. After looking into it, we discovered there was not much to do about this besides sit and wait… We do wish to apologize and clarify the events for this outage (our first since July 2012).

Windows Azure Storage outageThe root cause of this outage was a global outage of Windows Azure Storage. This service is one of the code building blocks of Windows Azure and has never in the past failed (that’s 4 years of no issues). This storage system works based on the HTTP protocol and has both http and https endpoints. Most applications built on top of Windows Azure, including the platform’s own building blocks, are using the https endpoints to prevent transport-level attacks, including MyGet. Unfortunately, most clusters of Windows Azure Storage were running an expired SSL certificate on this https endpoint, the reason for this global outage of Windows Azure and every application hosted on the platform, including MyGet and the official NuGet package source and every service directly depending on www.nuget.org.

MyGet runs in the Windows Azure Europe West region (Amsterdam), with a cold disaster recovery location in the Windows Azure Europe North region (Dublin). We can fail over to this location within hours if compute or storage in the main datacenter location fail. In case of a serious outage, we can restore a disaster recovery copy of our services in any Windows Azure region around the globe. Unfortunately, there’s nothing much we can do in case of a global outage…

Our status page can always be found at http://status.myget.org. For reference, here are our uptime numbers for the past year.

Name

Uptime
(avg.: 99.79%)

February 2013

98,44%

January 2013

100%

December 2012

99,99%

November 2012

99,94%

October 2012

99,99%

September 2012

99,92%

August 2012

99,98%

July 2012

99,56%

June 2012

99,94%

May 2012

100%

April 2012

99,69%

March 2012

99,99%

 

Again, we do apologize for the inconvenience caused and are debating around possible fallback scenarios in case a severe platform outage like this occurs again.

Happy packaging!

Author: Maarten Balliauw on 24 Feb 2013

Introducing activity streams

In a previous post we highlighted the new user profile page and briefly mentioned we are working on activity streams. If you look at the Activity tab of your profile, you'll still see that it is coming soon.

We are still working on the UI for this part, but before we introduce it to the masses we need to take care of a few more things. No worries, a clear communication about these changes will be done upfront, and we obviously will keep private feed activity secured. A lot of stuff is happening under the hood, really, in order for this feature to be rolled out smoothly. Our incremental and short release cycles are starting to pay off (or you'd end up with blank activity streams upon release).

However, you might notice that we already built an initial UI for the feed activity. Simply take a look at your feed details page and check the Feed Activity tab on the left menu.

We are working hard on providing you with historic information on packages and feeds throughout the site so expect more to come soon. We believe this will greatly improve the experience of both package publishers as consumers, as well as those interested to keep up with what's happening on the feed or with a specific package. There's some great stuff on some of the feeds in our Gallery and we'd like to give you the opportunity to get notified of the package repository events you want to track.

The below screenshot also gives you an idea of the kind of activity you might be interested in: adding or deleting packages, unlisting packages, pushing packages to an upstream package source (e.g. to NuGet.org), etc. The particular feed activity you can see below is the result of how we push the GoogleAnalyticsTracker package to NuGet.org.

Happy New Year & Happy Packaging!

Author: Maarten Balliauw on 07 Jan 2013

Cloning feeds is now supported

It happens that for some reason you require a full copy of an existing feed. You may want to do some upgrades. Maybe you just require your development feed to be copied as a production feed or vice-versa. Our latest deployment provides you with a fresh feature: cloning feeds.

From the feed list (www.myget.org/feed/list), simply click the “Clone” button next to a feed. Note that this will only be shown for feeds that are owned by you.

image

After a couple of minutes, your feed clone will be up and running.

image

Happy packaging!

Author: Maarten Balliauw on 25 Dec 2012