MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

GitHub Commit Status API now supported

A while ago, GitHub launched their Commit Status API which allows integrating the status of a build with commits and pull requests on GitHub. When a MyGet build source is linked to a GitHub repository and has credentials specified, we have started reporting status messages back to GitHub.

GitHub Commit Status API MyGet

When a build succeeds or fails, you will see a status message posted to GitHub, linking to the build log on MyGet.

MyGet reports build status to GitHub

To enable GitHub Commit Status messages on your builds, make sure the build configuration has credentials specified. Specifying credentials can be done by removing and adding the build configuration again, a method which doesn’t require you to enter your password. You can also specify credentials manually by editing the build source.

Happy packaging!

NuGet Package Restore and MyGet Build Services

With the release of NuGet 2.7, a new way of doing package restore came to life. Package restore allows you to keep only your source code and packages.config under source control and just download and install NuGet dependencies during build.

What does this mean for MyGet Build Services? Let’s say we’re making it easier for you! When building from a Visual Studio solution or project, there is nothing you should do: we will run package restore automatically. Let’s dive into the package restore conventions we have in place. Full details on the build process are available from our documentation.

MyGet Build Services

Package Restore Conventions

MyGet Build Services runs NuGet Package Restore as part of every build of solution or project files even if it's not enabled for the solution. Note that package restore is not run for builds making use of batch or PowerShell scripts. In those cases, you are the responsible for running package restore.

In order of precedence, the following package restore commands are run. When one succeeds, package other commands will be skipped.

  • nuget restore MyGet.sln -NoCache -NonInteractive -ConfigFile MyGet.NuGet.config
  • nuget restore MyGet.sln -NoCache -NonInteractive -ConfigFile NuGet.config
  • nuget restore <your solution file> -NoCache -NonInteractive -ConfigFile MyGet.NuGet.config
  • nuget restore <your solution file> -NoCache -NonInteractive -ConfigFile NuGet.config
  • nuget restore packages.config -NoCache -NonInteractive -ConfigFile MyGet.NuGet.config
  • nuget restore packages.config -NoCache -NonInteractive -ConfigFile NuGet.config
  • nuget restore MyGet.sln -NoCache -NonInteractive
  • nuget restore <your solution file> -NoCache -NonInteractive
  • nuget restore packages.config -NoCache -NonInteractive

If you are working with other feeds than the default NuGet.org feed, there are some options available.

Restoring from other Package Sources

If you want MyGet Build Services to restore packages from a specific feed, there are several options available to do this. The easiest way of making a package source available to the build process is by adding it through the UI.Making package source available during build

Using the Add package source button, you can add any feed that you want to make available during build: a Chocolatey feed, a TeamCity feed or another MyGet feed. It is even possible to store credentials so an authenticated feed can be used during build.

If you prefer to have all feed information in your source repository, that’s an option too. Adding a MyGet.NuGet.config file to your repository is the key to success. See the NuGet docs for more information on how such file can be created. The following is a sample registering a custom NuGet feed for package restore.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="Chuck Norris feed" value="https://www.myget.org/F/chucknorris/api/v2/" />
  </packageSources>
  <activePackageSource>
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

 

A small remark: if you have an authenticated feed but do not wish to add credentials to source control, credentials can be added to the feed's package source. These credentials will be available during build and allow you to consume a protected feed with ease. Authentication of feeds

Summary

Using NuGet Package Restore in MyGet Build Services is a breeze. We follow standard NuGet conventions to do your builds and allow adding package sources which will be made available during build through the UI. If you do need to customize things, there are several hooks available too. Full details on the build process are available from our documentation.

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

Retention Rules and Package Pinning

When using MyGet feeds for package archiving or nightly builds, a lot of packages show up on the feed very quickly. By default, we keep all package versions available on your feed. If you would like to do some automated housekeeping, retention rules can be added per feed. Whenever a package is added to your feed, we'll make sure these retention rules are respected.

Package retention rules for a feed

Our latest release added some additional options for package retention. You can now:

  • Specify the number of stable versions to keep
  • Specify the number of prerelease versions to keep
  • Specify if we have to keep depended packages or not

That last one is different from what we did in the past. We used to blindly delete packages that were older than version X, even if there were packages depending on it. From now one, we will keep packages that are depended on unless specified through the settings.

Pinning Package Versions

Certain packages should never be deleted automatically. For every package, we now support pinning specific versions – a pinned package will never be deleted by the retention rules. From the package details, this can be done using the pin/unpin buttons.

Pinning packages - ignore retention rules

Let us know what you think about this feature through the comments below or in our forums!

Happy packaging!

Release notes for MyGet 1.8

MyGet 1.8 was released on September 10, 2013. We will blog about new features in the next days and weeks.

Features

MyGet

  • Support for NuGet 2.7
  • Metadata for packages is auto-updated from upstream feeds
  • Retention policies: pin packages so they don't get deleted
  • Retention policies: packages that are depended on will no longer be deleted (unless explicitly enabled)
  • Push upstream: package source code repositories can be labeled when pushing packages upstream
  • Send e-mail when feed permissions change
  • Users can revoke their own access from a feed
  • Automatic mirroring for packages from upstream feeds when feed proxy is enabled

MyGet Enterprise

  • Administrators can now join a feed. Feed owners are notified of this action.

MyGet Build Services

  • Repositories from GitHub organizations are now shown
  • The latest build version is shown in the UI
  • Package sources added at the feed level are available on the build server
  • Automatic support for NuGet package restore even if it's not enabled for the solution
  • Support for NuGet 2.7 package restore - see http://docs.myget.org/docs/reference/build-services#Package_Restore
  • Package sources added at the feed level are available on the build server for package restore
  • Build labeling: on succesful or failed builds, a label can be added to the sources. This is compatible with GitHub Releases. - see http://docs.myget.org/docs/reference/build-services#Sourcelabeling(tagging)
  • Support for MyGet.cmd, MyGet.bat, MyGet.ps1

Bug Fixes

  • Fixed an issue with copy to clipboard
  • Fixed an issue when pushing packages upstream to authenticated feeds
  • Support page is no longer behind an authentication wall
  • Fixed an issue when build services used @ in the username
  • Various performance improvements

Happy packaging!

Downtime September 11 and 12, 2013 - Root cause

On September 11 and 12, 2013, we have experienced some downtime. The website and back-end services have crashed a number of times across all instances, bringing our system to a halt a couple of times. We hate downtime as much as you do and want to apologize for the trouble this may have caused you. Let’s have a look at the symptoms and root cause.

Symptoms

Twitter and e-mail notified us of this a good 2 minutes before our monitoring kicked in. When we had a look at our status dashboards, we saw high CPU usage on all machines, inexplicable crashes of machines and an error message in the Windows event logs right before the machines died and restarted.

In our back-end queue, where all work is added and waits to be processed, we found 25 messages that seemed to be retried over and over again, having been dequeued over 150 times.

We have manually removed these messages from the queue and inserted them again after the website was running stable again. After monitoring the situation for a couple of hours, all systems seemed to be running stable again. Seemed…

A number of hours later, we received some additional monitoring events. The problem was back! This time we decided not to fire fight but dig deeper into the issue we were facing. We took the site offline and connected to the machines to analyze event logs and so on. Except for the error message in the Windows event log, nothing was going on. Next, hooked a debugger to it, waiting for anything to happen.

The issue

Perhaps a root cause analysis is not the best moment to blog about our new features and updates in our latest deployment, but we have to in this case. Last Tuesday, we deployed our 1.8 release which brings an update to retention policies. These allow MyGet to automatically delete packages from a feed after a certain number of packages has been added, an automatic feed cleanup as it were.

The updated retention policies feature now tries not to break a dependency chain. Before, packages that were depended on by other packages but were subject to the retention policy settings would be deleted. Now, the retention policy handler respects the entire dependency chain for a package and will no longer remove depended packages. This, of course, requires us to parse the entire dependency tree for a package.

imageNow what happens if package A depends on package B, B depends on C and C depends on B? Right: that’s a potential infinite loop. Guess what was happening: a StackOverflowException, crashing our entire machine without giving it the chance to write output to our error logs.

Because of the crash, the message in our backend was requeued over and over. We typically move a message to a dead letter queue after a number of retries (typically 32). However because of the crash, that logic couldn’t kick in and move the message out, resulting in the message being requeued over and over again, triggering the same behavior again.

While we test our application thoroughly before going to production, this was an edge case. Who would create a package with circular dependencies anyway? We found that on one feed, a circular package dependency did exist.

Solution

We have made a fix to our algorithm which now supports this scenario and will stop following the circular dependency.

After confirming our algorithm, we have deployed it at 6 AM (CET) this morning, resolving the issue. The messages that were stuck in our queue (and triggered the initial issue) have been requeued to confirm correct working and  were all processed successfully.

Our status page can always be found at http://status.myget.org, showing uptime of our last months as well as the outage of the past hours.

Again, we do apologize for the inconvenience caused.

Happy packaging!

Online gaming with OCTGN and NuGet

OCTGN online card and table game networkWe’ve been looking through some of the feeds that are hosted on MyGet and have found a nice use case for both NuGet and MyGet: the open-source project OCTGN, the Online Card and Table Game Network. As a result of that, we’ve had a chat with Kelly Elton, developer and maintainer of OCTGN.

Can you tell us a little bit about OCTGN?

“The title of the game kind of summarizes this, but OCTGN is a gaming platform that allows users to play card and board games with each other on the Internet. After installation, gamers get a virtual table top where they can download additional games from a repository, chat with other gamers and compete against each other.”

We’re assuming that the games repository is in fact a NuGet feed, is that correct?

Yeah, that’s an interesting one. Started using NuGet a while ago and it occurred to me that you could use NuGet as an application update system. It has metadata in there which you can query and the NuGet.Core assembly gives you a nice library for working with it.

Originally, our users would download an OPC (Open Packaging Conventions) file which is essentially a ZIP file. A NuGet package essentially also is an OPC package, so we were already using some form of packaging for distributing games before. And then we did some rework and tore some pieces apart and realized we could start packaging using NuGet to get things like searchable metadata, a formal upgrade system and so on.”

A lot of the games are card games, are you embedding the card artwork in the NuGet packages as well?

“No. Sets of cards can be 100 MB to 200 MB in size. Not all players will use all cards all the time and we don’t want game updates to download all artwork when it hasn’t changed between versions. Also we have a card proxy which generates images from card data on the fly if they don't exist, which allows us to cram a game that used to require 100MB of images to play into a 10MB package, with the option to download the cards from another source if they so choose.

We developed a system that allows you to specify the metadata for the cards which is kind of a specialized markup language and then download the artwork on-the-fly, only when needed.”

Who is developing the games for OCTGN? How can someone contribute?

In-game screenshot”Most of the games are developed by community members. OCTGN has been around since 2006 and has changed hands a few times but it’s essentially run by the community. I’m currently doing most of the development but I have some people helping me. The games themselves are developed by people, using Python and XML. We have a builder (o8build) which comes with OCTGN, takes the Python code and XML files and packages everything up so our game developers don’t have to know too much about NuGet themselves. There’s some info about that on our GitHub space.

Contributions are done in several ways. We have different NuGet feeds that are managed by game owners, we also have our official feeds that contain games that we want to be part of our system. You get added to that feed based on different metrics like game quality, how active development is and so on. There are different feeds maintained by other members of the community and they can take on some games as well.

We only host the official feed, but since it's so easy to get setup with MyGet, people in the community have had no problems at all setting up their own feeds.

Are all feeds public feeds?

“Some of our game developers asked if it would be possible to make a private feed for their games, we’re now looking into adding support for that in our client. We weren’t sure how we could authenticate the user towards these private feeds as the NuGet client libraries aren’t that well documented. But I’ve seen there is an ICredentialProvider interface in there we can probably use.”

How has your experience with MyGet been so far?

“Very good, we’ve enjoyed it quite a bit. We didn’t quite know if it was going to be the right system for the job at first as we were almost going to abstract away the way packages were offered to the game. Our big concern was that we have almost 40.000 registered users and we didn’t want to step on your toes with regards to package sizes, bandwidth consumed and so on. We wanted to make sure we weren’t causing any cause of problems as a free user of MyGet.

As far as quality goes we have absolutely had no problems with MyGet at all. It’s really easy to upload and manage packages and we’re now looking into using the build system you have as that’s pretty neat.”

As a last question, do you have any advice for people starting with NuGet or MyGet?

“For MyGet not so much as it can’t be more straightforward. Even people that didn’t know what NuGet was were able to setup feeds and upload their games in a matter of a minutes. That was really easy and I would encourage people to try it out. You guys are doing some awesome stuff!

One thing I’ve noticed with NuGet that tripped people up is versioning, sometimes the version numbers weren’t sorted in the way we’ve expected but we’ve tackled that. I believe Xavier had a blog post about this.

Another thing is most people seem to be focused on creating or consuming packages and that is also what all the documentation on the NuGet site is about. Once you realize that NuGet can also be used as a distribution system of packages for your own application, things get more interesting. You can get started developing such thing by adding the NuGet.Core assembly to your project and navigating around with IntelliSense. The whole system is pretty slick and I’m using it in other applications I work on as well.

NuGet has a lot more uses than what can be seen at first sight, there is a whole lot to explore!”


Thank you Kelly for this chat, really appreciated!

Just like OCTGN, if you are working on an open source project you can get an Open Source subscription plan for your project!

Happy packaging!

How SharpDevelop uses MyGet and NuGet

This is a guest post by Matt Ward, working on SharpDevelop, an open source Integrated Development Environment (IDE) for .NET applications which supports the development of applications written in C#, Visual Basic.NET, F# and IronPython. It is and written in C#. In this post, Matt shares their story on using MyGet to host and discover add-ins for SharpDevelop.

If you would like to share your story too, let us know! We love to hear good stories!

SharpDevelop NuGet MyGetThe SharpDevelop team recently announced a new Addin Manager for SharpDevelop 5 and an online gallery for addins that is hosted on MyGet.

SharpDevelop has been extensible through addins since it was first created but it has never before had an online gallery. The majority of IDEs have an online gallery that can be used to find and install extensions. WebMatrix is one example that provides extensions through from its own NuGet feed just like SharpDevelop.

Addins for SharpDevelop 5 can now be published as NuGet packages to a MyGet gallery. NuGet gives SharpDevelop all the benefits of versioning and updating addins whilst MyGet provides a central place where SharpDevelop addins can be found and installed from.

SharpDevelop's addin gallery on MyGet is a community gallery which is open to anyone to publish their own addins.

Background

About a year ago there was a discussion on the SharpDevelop mailing list about having an online source of addins for SharpDevelop. Andreas Weizel had started work on a way to obtain addIns from an online source but at the time it was not based on NuGet. Christoph Wille, Project Manager for SharpDevelop, suggested looking at re-using the NuGet gallery for hosting the packages and using NuGet to download and install addins. One idea was to put the addins on the main NuGet.org feed and add a special tag to them to distinguish them however it made more sense for SharpDevelop to have its own gallery. Rather than setting up a server and hosting a NuGet gallery ourselves it was quicker and easier to use MyGet.

Now let us take a look at how you can make your own addins available for SharpDevelop 5. If you have published a NuGet package to the gallery hosted on NuGet.org then the following procedure will be very familiar.

Publishing an Addin with MyGet

Here we will look at the steps taken to publish an addin that provides support for compiling against Mono. The source code for this sample addin is available on GitHub. The first step is to compile the addin. With the Mono addin compiled we need to create a .nuspec file that will be used to generate a NuGet package containing all the files needed for the addin:

    <?xml version="1.0"?>
    <package xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <metadata xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
            <id>Mono</id>
            <version>1.0</version>
            <authors>Matt Ward</authors>
            <owners>SharpDevelop</owners>
            <requireLicenseAcceptance>false</requireLicenseAcceptance>
            <description>Mono support for SharpDevelop</description>
            <iconUrl>http://community.sharpdevelop.net/blogs/mattward/SharpDevelop.png</iconUrl>
            <summary>Mono support for SharpDevelop</summary>
            <releaseNotes></releaseNotes>
            <language>en-US</language>
            <tags>mono</tags>
        </metadata>
        <files>
            <file src="..\..\..\AddIns\Samples\Mono.Addin\**\*.*" target="" />
        </files>
    </package>

In the above .nuspec file we are adding all the addin files, including any subdirectories, by using the double wildcard ** so we do not have to explicitly specify every file. SharpDevelop requires the main addin assembly and its associated .addin configuration file to be in the root of the NuGet package. This is why the target for these files defined in the .nuspec file has been left empty.

Now we can generate the NuGet package by running NuGet.exe from the command line:

nuget pack mono.nuspec

On running this command you will see a "Assembly outside lib folder" warning which can safely be ignored since we are not going to be adding this NuGet package to a .NET project. We have now generated a Mono.1.0.nupkg file which is what we will publish to the MyGet gallery.

To publish to MyGet you will need your MyGet account's personal API key. Your API key can be found under Account Settings in the Security section. Copy your API key and in the following command line replace the Your-API-Key with your key:

Running the above command will publish the Mono addin to MyGet. Now anybody using SharpDevelop 5 can install the Mono addin by selecting AddIn Manager from the Tools menu.

image

You can see the list of available addins by clicking the Available section of the dialog. To install the Mono Addin select it and click the Install button. The addin will be available after SharpDevelop is restarted.

Matt Ward is developer at Pebble Code. In his spare time he works on SharpDevelop, which he has been involved with since 2004, and more recently has been working on a NuGet addin for MonoDevelop and Xamarin Studio.

Blog: community.sharpdevelop.net/blogs/mattward Twitter: @sharpdevelop

Automatically mirror packages from other feeds (or: keep working during NuGet outages)

imageMany of our users have created their own NuGet feed on MyGet and are uploading their own packages to that hosted feed. Some of our users have been asking for a good way to make their MyGet feed the only feed in their team, the reason for introducing MyGet package source proxy which can include packages from upstream package sources such as the official NuGet Gallery in search results.

Up until recently, package source proxies were nothing but proxies: the feature simply proxies the packages on demand, which means a number of things:

  • Upstream packages that are deleted are no longer available on your MyGet feed unless explicitly mirrored.
  • When an outage occurs on the upstream package source, the proxy is of no use because it can’t connect to the upstream package source

Today, we are introducing a solution for that: a new option is available to automatically add and mirror downloaded upstream packages to your feed, as requested in these two UserVoice entries.

Enabling this option ensures that you can keep working with just one feed and have all the packages you need available on that feed, even if they originate from another package source and even if that upstream package source happens to have an outage. If you are working with firewall exceptions, enabling this option will also ensure that only one firewall exception to your MyGet feed has to be made.

If you want more information on how to enable this feature, check out an end-to-end tutorial in our documentation.

Happy packaging!

MyGet is running over HTTPS only

A couple of weeks ago we've informed you that from July 1st, 2013, MyGet would be switching to HTTPS traffic only. We have just deployed these changes and from now on, your feed URL should be prefixed with https:// instead of http://.

Haven't made the switch yet? No problem: we will be redirecting most traffic to the new HTTPS endpoint. However this may add a little latency to your requests. Hence it's best to switch your URL now if you haven't done so.

To help in updating feed URLs on developer machines, you can make use of package source discovery. http://docs.myget.org/docs/reference/package-source-discovery
In short, every developer can issue the following commands in his/her Visual Studio Package Manager Console to update feed URLs:

Install-Package DiscoverPackageSources
Discover-PackageSources -Url https://www.myget.org/Discovery/Feed/ -OverwriteExisting

Protecting the security and privacy of our users is one of our most important tasks at MyGet. The fact that you can safely store your intellectual property on our servers is the best proof of that. We are confident this one-time change will make the entire MyGet experience even more secure.

Do not hesitate to contact us if you have additional questions through the comments below.

Happy packaging!

PS: If you are a MyGet Enterprise customer and are using your own domain name, no action isrequired on your side.

Build services: patching AssemblyVersion attribute

It already was possible to work with true incremental build numbers for packages produced using Build Services through the build source settings. A build counter starts with zero and increments with 1 on every build. You can also specify a version format (use '{0}' as a placeholder for the build counter) which will be generated during build.

Build Services recently got an update where the AssemblyVersion attribute can be patched with this version number. This can be enabled by checking the Automatically patch AssemblyInfo option in the build source configuration.

Patching AssemblyVersion during build

When enabled, MyGet Build Services will patch AssemblyVersion attributes in C# and VB.NET code. We are using Roslyn as the engine for parsing and updating attribute values. This approach is much more reliable than the regular expression based approaches most build systems use.

Two attributes will be patched: AssemblyVersion and AssemblyInformationalVersion.

  • The patched AssemblyVersion version is always in the form major.minor.patch. A package version 1.0.0 as well as 1.0.0-pre will yield an AssemblyVersion of 1.0.0.
  • The patched AssemblyInformationalVersion version supports semantic versioning and can be in the form major.minor.patch as well as major.minor.patch-prerelease.

Patching of these attributes will occur whenever the feature is enabled, no matter which build process is used (solution, project or build.bat).

Happy packaging!