MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

GitHub Releases and MyGet Build Services

During the summer, GitHub added a great feature for many projects: releases. Projects on GitHub can create a release and label sources, add binaries and release notes. Following the conventions of many Git projects, releases are tied to Git tags. You can use an existing tag, or let releases create the tag when it's published.

Since MyGet Build Services supports labeling and we can label sources when pushing packages upstream, GitHub Releases make a great combination with MyGet! Let’s see.

Labeling Sources from MyGet

When enabled in the build source configuration on MyGet, source code can be labeled with the build number. This can be done for successful builds only (recommended) as well as for failed builds.

Label Sources during Build and make it a GitHub release

It is also possible to base tags for GitHub Releases on having a fresh package pushed to, say, NuGet.org. This can be done while pushing packages upstream or “promoting” them. A dialog will provide you with additional options, e.g. configure the package version to be used upstream, as well as the option to label the sources for packages originating from MyGet Build Services.

GitHub release when pushing to NuGet.org

Releasing through GitHub

Regardless of the approach taken for creating a label, all that is left now is creating the release on GitHub. Since either the build process or the package promotion process have created the label already, all that GitHub requires is a set of release notes and a description for the release.

Integrating GitHub Releases and MyGet

Pretty sweet, no? Let us know your thoughts through the comments below or in our forums!

Happy packaging and shipping!

Labeling Sources when Pushing to NuGet.org

When adding package sources through your feed’s settings, a very nice scenario becomes available: the package promotion workflow. In other words: pushing a package from one feed to another. Or in other words: publishing nightlies to MyGet and promoting specific package versions to NuGet.org.

With the newly introduced labeling feature, it is now possible to label sources when pushing a package upstream. When enabled, MyGet will find the build from which the package originated and will add a label to the source control revision it was built from. Note that the build must originate from MyGet Build Services for this to work.

Choose the package you want to promote and with a click of a button you can push it upstream. A dialog will provide you with additional options, e.g. configure the package version to be used upstream.

Label sources when pushing upstream

An important note: If you want to make use of labeling, you will have to specify credentials to connect to the remote repository, or remove and add the build source again. Labeling will fail if this is neglected.

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

Using additional tools in the build process

Out of the box, MyGet Build Services supports building projects that depend on various SDK's and/or test runners. A full list of supported project types and SDK's at your disposal is available on our docs.

However, sometimes you run into a missing tool or SDK you absolutely need to compile your projects. Or perhaps you need a different version than the one available by default. If that's the case, you have two options:

  • Download the tools you need and commit them to your own repository
  • Use the brand new MyGet Build Tools repository on GitHub as a submodule to your repository

The first option is likely what you're doing today and allows you to cherry-pick the required tools without polluting your build with tools you don't need, making the build process faster.

The second option will bring down the tools from this repository during a build while they are technically not in your own GitHub repository. It currently contains the following tools:

  • NuGet 2.5, 2.6, 2.7, 2.7.1
  • NUnit 2.6.2
  • XUnit 1.9.6
  • curl

To add a submodule, run the following from your repository root (make sure you use the https endpoint):

git submodule add https://github.com/myget/BuildTools.git myget

From then on, you can use the tools from the myget/tools folder in your builds. For example, NuGet 2.5 can be used by calling myget/tools/nuget/2.5/nuget.exe.

Happy Packaging!

Making your life easier with multiple access tokens

What if you are using your API key on your TeamCity server and several other locations and for some reason you have to reset that API key? Major headache? Not anymore: from now on, MyGet supports multiple access tokens.

Since MyGet day one, we’ve had support for two credentials linked to your account. The primary API key could be used when publishing packages with NuGet.exe or NuGet Package Explorer while your username and password were useful when consuming private feeds from Visual Studio or a build server.

Additional access tokens can be generated from your profile page. The primary API key can be regenerated and new tokens can be easily created or revoked. Access tokens can be given a short description: this will help keeping track of where you used the access token and revoke it if necessary.

Access tokens can be used for all authentication purposes. They can be used when pushing to your MyGet feed or as an alternate password when authenticating against a private feed or SymbolSource.org.

Go ahead and separate the credentials you use personally, on the build server or in other software. Access tokens can be created and revoked at any time.

image

Happy packaging!

Labeling Sources after Build

When enabled in the build source configuration on MyGet, source code can be labeled with the build number. This can be done for successful builds only (recommended) as well as for failed builds.Labeling Source Code from MyGet

The label originating from MyGet will always be named in the form vX.Y.Z, vX.Y.Z.P or v.X.Y.Z-pre. The description for the label will always be the label name (the version number), followed by "- MyGet Build Services". This labeling scheme is compatible with GitHub releases and can link a given NuGet package version number to a GitHub release.

An important note: If you enable build labeling on build configurations that have been created before 2013-09-11, you will have to specify credentials to connect to the remote repository, or remove and add the build source again. Builds with labeling enabled will fail if this is neglected.

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

GitHub Commit Status API now supported

A while ago, GitHub launched their Commit Status API which allows integrating the status of a build with commits and pull requests on GitHub. When a MyGet build source is linked to a GitHub repository and has credentials specified, we have started reporting status messages back to GitHub.

GitHub Commit Status API MyGet

When a build succeeds or fails, you will see a status message posted to GitHub, linking to the build log on MyGet.

MyGet reports build status to GitHub

To enable GitHub Commit Status messages on your builds, make sure the build configuration has credentials specified. Specifying credentials can be done by removing and adding the build configuration again, a method which doesn’t require you to enter your password. You can also specify credentials manually by editing the build source.

Happy packaging!

NuGet Package Restore and MyGet Build Services

With the release of NuGet 2.7, a new way of doing package restore came to life. Package restore allows you to keep only your source code and packages.config under source control and just download and install NuGet dependencies during build.

What does this mean for MyGet Build Services? Let’s say we’re making it easier for you! When building from a Visual Studio solution or project, there is nothing you should do: we will run package restore automatically. Let’s dive into the package restore conventions we have in place. Full details on the build process are available from our documentation.

MyGet Build Services

Package Restore Conventions

MyGet Build Services runs NuGet Package Restore as part of every build of solution or project files even if it's not enabled for the solution. Note that package restore is not run for builds making use of batch or PowerShell scripts. In those cases, you are the responsible for running package restore.

In order of precedence, the following package restore commands are run. When one succeeds, package other commands will be skipped.

  • nuget restore MyGet.sln -NoCache -NonInteractive -ConfigFile MyGet.NuGet.config
  • nuget restore MyGet.sln -NoCache -NonInteractive -ConfigFile NuGet.config
  • nuget restore <your solution file> -NoCache -NonInteractive -ConfigFile MyGet.NuGet.config
  • nuget restore <your solution file> -NoCache -NonInteractive -ConfigFile NuGet.config
  • nuget restore packages.config -NoCache -NonInteractive -ConfigFile MyGet.NuGet.config
  • nuget restore packages.config -NoCache -NonInteractive -ConfigFile NuGet.config
  • nuget restore MyGet.sln -NoCache -NonInteractive
  • nuget restore <your solution file> -NoCache -NonInteractive
  • nuget restore packages.config -NoCache -NonInteractive

If you are working with other feeds than the default NuGet.org feed, there are some options available.

Restoring from other Package Sources

If you want MyGet Build Services to restore packages from a specific feed, there are several options available to do this. The easiest way of making a package source available to the build process is by adding it through the UI.Making package source available during build

Using the Add package source button, you can add any feed that you want to make available during build: a Chocolatey feed, a TeamCity feed or another MyGet feed. It is even possible to store credentials so an authenticated feed can be used during build.

If you prefer to have all feed information in your source repository, that’s an option too. Adding a MyGet.NuGet.config file to your repository is the key to success. See the NuGet docs for more information on how such file can be created. The following is a sample registering a custom NuGet feed for package restore.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="Chuck Norris feed" value="https://www.myget.org/F/chucknorris/api/v2/" />
  </packageSources>
  <activePackageSource>
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

 

A small remark: if you have an authenticated feed but do not wish to add credentials to source control, credentials can be added to the feed's package source. These credentials will be available during build and allow you to consume a protected feed with ease. Authentication of feeds

Summary

Using NuGet Package Restore in MyGet Build Services is a breeze. We follow standard NuGet conventions to do your builds and allow adding package sources which will be made available during build through the UI. If you do need to customize things, there are several hooks available too. Full details on the build process are available from our documentation.

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

Retention Rules and Package Pinning

When using MyGet feeds for package archiving or nightly builds, a lot of packages show up on the feed very quickly. By default, we keep all package versions available on your feed. If you would like to do some automated housekeeping, retention rules can be added per feed. Whenever a package is added to your feed, we'll make sure these retention rules are respected.

Package retention rules for a feed

Our latest release added some additional options for package retention. You can now:

  • Specify the number of stable versions to keep
  • Specify the number of prerelease versions to keep
  • Specify if we have to keep depended packages or not

That last one is different from what we did in the past. We used to blindly delete packages that were older than version X, even if there were packages depending on it. From now one, we will keep packages that are depended on unless specified through the settings.

Pinning Package Versions

Certain packages should never be deleted automatically. For every package, we now support pinning specific versions – a pinned package will never be deleted by the retention rules. From the package details, this can be done using the pin/unpin buttons.

Pinning packages - ignore retention rules

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

Release notes for MyGet 1.8

MyGet 1.8 was released on September 10, 2013. We will blog about new features in the next days and weeks.

Features

MyGet

  • Support for NuGet 2.7
  • Metadata for packages is auto-updated from upstream feeds
  • Retention policies: pin packages so they don't get deleted
  • Retention policies: packages that are depended on will no longer be deleted (unless explicitly enabled)
  • Push upstream: package source code repositories can be labeled when pushing packages upstream
  • Send e-mail when feed permissions change
  • Users can revoke their own access from a feed
  • Automatic mirroring for packages from upstream feeds when feed proxy is enabled

MyGet Enterprise

  • Administrators can now join a feed. Feed owners are notified of this action.

MyGet Build Services

  • Repositories from GitHub organizations are now shown
  • The latest build version is shown in the UI
  • Package sources added at the feed level are available on the build server
  • Automatic support for NuGet package restore even if it's not enabled for the solution
  • Support for NuGet 2.7 package restore - see http://docs.myget.org/docs/reference/build-services#Package_Restore
  • Package sources added at the feed level are available on the build server for package restore
  • Build labeling: on succesful or failed builds, a label can be added to the sources. This is compatible with GitHub Releases. - see http://docs.myget.org/docs/reference/build-services#Sourcelabeling(tagging)
  • Support for MyGet.cmd, MyGet.bat, MyGet.ps1

Bug Fixes

  • Fixed an issue with copy to clipboard
  • Fixed an issue when pushing packages upstream to authenticated feeds
  • Support page is no longer behind an authentication wall
  • Fixed an issue when build services used @ in the username
  • Various performance improvements

Happy packaging!

Downtime September 11 and 12, 2013 - Root cause

On September 11 and 12, 2013, we have experienced some downtime. The website and back-end services have crashed a number of times across all instances, bringing our system to a halt a couple of times. We hate downtime as much as you do and want to apologize for the trouble this may have caused you. Let’s have a look at the symptoms and root cause.

Symptoms

Twitter and e-mail notified us of this a good 2 minutes before our monitoring kicked in. When we had a look at our status dashboards, we saw high CPU usage on all machines, inexplicable crashes of machines and an error message in the Windows event logs right before the machines died and restarted.

In our back-end queue, where all work is added and waits to be processed, we found 25 messages that seemed to be retried over and over again, having been dequeued over 150 times.

We have manually removed these messages from the queue and inserted them again after the website was running stable again. After monitoring the situation for a couple of hours, all systems seemed to be running stable again. Seemed…

A number of hours later, we received some additional monitoring events. The problem was back! This time we decided not to fire fight but dig deeper into the issue we were facing. We took the site offline and connected to the machines to analyze event logs and so on. Except for the error message in the Windows event log, nothing was going on. Next, hooked a debugger to it, waiting for anything to happen.

The issue

Perhaps a root cause analysis is not the best moment to blog about our new features and updates in our latest deployment, but we have to in this case. Last Tuesday, we deployed our 1.8 release which brings an update to retention policies. These allow MyGet to automatically delete packages from a feed after a certain number of packages has been added, an automatic feed cleanup as it were.

The updated retention policies feature now tries not to break a dependency chain. Before, packages that were depended on by other packages but were subject to the retention policy settings would be deleted. Now, the retention policy handler respects the entire dependency chain for a package and will no longer remove depended packages. This, of course, requires us to parse the entire dependency tree for a package.

imageNow what happens if package A depends on package B, B depends on C and C depends on B? Right: that’s a potential infinite loop. Guess what was happening: a StackOverflowException, crashing our entire machine without giving it the chance to write output to our error logs.

Because of the crash, the message in our backend was requeued over and over. We typically move a message to a dead letter queue after a number of retries (typically 32). However because of the crash, that logic couldn’t kick in and move the message out, resulting in the message being requeued over and over again, triggering the same behavior again.

While we test our application thoroughly before going to production, this was an edge case. Who would create a package with circular dependencies anyway? We found that on one feed, a circular package dependency did exist.

Solution

We have made a fix to our algorithm which now supports this scenario and will stop following the circular dependency.

After confirming our algorithm, we have deployed it at 6 AM (CET) this morning, resolving the issue. The messages that were stuck in our queue (and triggered the initial issue) have been requeued to confirm correct working and  were all processed successfully.

Our status page can always be found at http://status.myget.org, showing uptime of our last months as well as the outage of the past hours.

Again, we do apologize for the inconvenience caused.

Happy packaging!