MyGet Blog

Package management made easier!

NAVIGATION - SEARCH

Package mirroring is now enabled by default

Any service can experience a brief moment of downtime. This is also true for any upstream package source you configure in MyGet. That's why we have the package mirroring feature: when uploading or adding a package to your feed, the added package (and its dependencies) are stored into MyGet's own storage system and they remain available in the event of an upstream service outage.

This package mirroring checkbox used to be disabled by default. Why? Because we wanted to save on storage costs. You might think we were being cheap here, but today we are hosting over 50Gb of packages and serving many of them multiple times a day! In the early days of MyGet, it made sense to simply reference the packages from eg. nuget.org and redirect you to the package URL upstream. This has saved us quite some storage, bandwidth and backup costs whilst bootstrapping (you know the Free plan isn't free for us, right?).

However, as we are maturing, we also gain new insights in how you are using MyGet so we can more effectively reduce friction in the way you work with packages. We measured that 95% of our users are mirroring their packages, so we are now reversing the logic for it. As of today, the checkbox will be checked by default!

A popular question we get is: why aren't you hosting an entire mirror of nuget.org? Answer: this is another one of those backlog items that has been sitting around for years, and we believe NuGet API v3 will greatly facilitate such scenario. One day :)

Happy packaging!

Announcing Visual Studio Online integration

We are really excited to announce that as of today, MyGet enables new scenarios for all users of Visual Studio Online! Microsoft just announced Visual Studio Online Extensions at TechEd, so we can finally disclose these new scenarios and toggle the feature-switch for all of you.

In short, these are the new scenarios we've just enabled for you:


New to MyGet? Interested in using the Visual Studio Online integration? Why not make use of our TechEd 2014 offer that gives you a complimentary 2-month Starter plan!

New Scenario: Serve NuGet packages from a Visual Studio Online drop location

Having a VSO Build Definition creating NuGet packages but having trouble consuming those? You can now very easily create a MyGet feed and serve those NuGet packages by having us fetching them from your Drop Location! Optionally, we can post notifications to your VSO Team Room so your team gets informed about new package availability as the result of your automated builds.

Package sources could be considered as upstream repositories for your MyGet feed, which is how you can look at the VSO Build Drop Location. The following simplified diagram provides a schematic overview of this new integration scenario.


All you need to do to make us of this integration is to add a new package source to your MyGet feed:

  1. Go to your feed's settings and select the Package Sources tab. Select Visual Studio Online build definition from the Add package source button.

  2. Provide your Visual Studio Online account name in the dialog that appears and click Continue.

  3. Select the Team Project and Build Definition to add as the package source for your feed. Optionally, select the Team Room in which to post notifications. Click the Add button to complete the configuration of the new package source.

  4. The new VSO package source has been added to your feed's package sources, and newly built NuGet packages will be fetched from your Drop Location automatically and pushed to the MyGet feed.

New Scenario: Add a VSO Git repository as a Build Source for a MyGet feed

This is the same scenario we already support today for GitHub, BitBucket and Codeplex: you can now register MyGet Build Services as a post-commit hook to your VSO Git repositories. We are happy to provide an integrated VSO experience from within MyGet, but even happier to also have a MyGet service hook available within the VSO user interface! Whether you use the VSO user interface or MyGet web site, you can enable this scenario from within both.

Build sources are a way to express links between MyGet Build Services and your source repositories, whether on GitHub, BitBucket, Codeplex or now also on Visual Studio Online. The following simplified diagram provides a schematic overview of this new integration scenario.

Note: we currently only support Git repositories, TFVC support is on the roadmap.


Here's a step-by-step guide on how to register a VSO Git repository within MyGet Build Services.

  1. Go to your feed's settings and select the Build Services tab. Select from Visual Studio Online (git only) from the Add build source… button.

  2. Provide your Visual Studio Online account name in the dialog that appears and click Continue.

  3. Select the desired build source to link and optionally enable notifications in a specific Team Room. Click the Add button to complete the configuration of the new Build Source.

  4. The new VSO build source has been added to your feed's build sources, and a post-commit service hook has been registered within VSO. When pushing to your VSO Git repository, a new MyGet build is going to be triggered and will produce your NuGet packages and push them to your feed.

    If you look at your Visual Studio Online Administration dashboard and browse your Team Project's Service Hooks, you'll see a newly registered web hook listening for the Code is pushed event.

New Scenario: Connect an existing MyGet Build Source to Visual Studio Online

Alternatively, you can also setup MyGet integration from within Visual Studio Online. This is useful if you already have an existing MyGet Build Source pointing to your VSO Git repository and triggered it manually up until now. You should be glad to hear you can now connect these and automate that.

  1. Go to your VSO Team Project Administration dashboard and select the Service Hooks tab.
  2. Click the Add Service Hook button

  3. Within the New Service Hooks Subscription dialog, select the MyGet service and click the Next button.

  4. Select the Code is pushed event type and the desired Repository and Branch settings. Click Next to continue.

  5. Select the Trigger a MyGet build action and provide the target Feed name and Build Source Identifier.

    You can find the Build Source identifier in your MyGet feed settings > Build Sources, as highlighted below:

  6. Click Test if you want to test this service hook first, or click Finish to complete the wizard. The new service hook is now listed as shown below:

We aim to please

We hope you are happy with this brand new VSO support and will put it to good use! We'd also like to express our gratitude to the Microsoft VSO team for enabling this type of extensibility as well as guiding us through this integration effort. A smooth, integrated user experience to reduce development friction is one of our main goals, and we hope these scenarios will prove to be a great help.

Feel free to send us your feedback or feature requests!

Happy packaging!

Using MyGet as a OneGet package source

At the Build conference, Microsoft announced the Windows Management Framework 5.0 Preview which includes Windows PowerShell 5.0, updates to the PowerShell ISE, Network Switch Cmdlets and ... OneGet!

What is OneGet?

OneGet a unified package management interface component with a set of managed and native APIs, a set of PowerShell cmdlets, and a WMI provider. The component accepts both Microsoft-provided and 3rd party-provided plugins which extend the functionality for a given package type.

The OneGet team also has a weekly community meeting of which you can see the first introductionary recording below.

As part of this Preview, OneGet is shipping with a prototype plugin compatible with Chocolatey, the so called ChocolateyProvider. This is a prototype implementation of a Chocolatey-compatible package manager that can install existing Chocolatey packages. This is a clear confirmation for the hard work done by the Chocolatey folks, and both systems will continue to evolve together, as Rob Reynolds explains in this post. If you want to follow-up on OneGet, then check out its GitHub repository and follow PSOneGet on Twitter.

Something about a forest and trees...

NuGet, MyGet, Chocolatey, OneGet... what?! People ask questions and occasionally can't see the forest for the trees. Here's a quick recap:

  • NuGet: a solution-level package management tool, used to manage software dependencies within the scope of a solution. It is accompanied by the NuGet Gallery, the home of many if not all .NET open source components.
  • Chocolatey: a system-level package management tool, used to manage software installations on a Windows system. It (currently) leverages PowerShell and NuGet, supports the Web Platform Installer (WebPI), MSI, RubyGems and many more, and is accompanied by the Chocolatey Gallery where you can find many popular software packages. Rob describes Chocolatey as somewhat like "apt-get", but with Windows in mind.
  • MyGet: a hosted NuGet package server where you can create and secure your own feeds. In essence, MyGet is able to host vanilla NuGet feeds, as well as Chocolatey feeds.
  • OneGet: a a unified interface to package management systems (see above)

So what does this mean? How do these package managers play along?

OneGet supports multiple package sources, and as stated earlier, OneGet comes with a ChocolateyProvider. As MyGet also supports Chocolatey feeds, this effectively means that you can register a MyGet feed as a Chocolatey package source in OneGet! The blow diagram is an attempt to illustrate how they relate:

How do OneGet, Chocolatey, NuGet and MyGet play along?

OneGet supports several commands at this stage:

OneGet Preview cmdlets

How can I use a private OneGet package source?

So how can I register a private OneGet package source? Well, let's first see how you can register any package source using the Add-PackageSource cmdlet. Here's what the built-in help currently says:

OneGet Add-PackageSource Help

Note that this is a Preview: help is incomplete and cmdlets might change name, but this should already give you a good idea of what you can do with this cmdlet!

Now, let's register a MyGet feed on which you can host Chocolatey packages:

Register a MyGet feed as a OneGet package source

Did you notice how OneGet asked you to install the NuGet package manager?

That went easy right? That's because that feed was public :) OneGet does not support basic-authentication at this point, nor does it leverage any nuget.config settings you might configure (tried it). However, MyGet just added the possibility to use a "private-by-obscurity" endpoint on private feeds, which should allow you to use private feeds as well. Note: we don't actively promote this, as it requires you to share one of your feed's access tokens. This is a work-around for clients that don't support the basic-auth flow, and we'd prefer to have proper basic-authentication support in OneGet, so fingers crossed!

You can verify the correct registration of your OneGet package source using the following commands:

Get OneGet package sources List available packages on a OneGet package source

Installing a software package from this MyGet feed is straightforward as well:

Install a software package from a specific OneGet feed

This flow allows you to control what packages get distributed through OneGet, avoids the need to publish your internal software to the general public, and still allows you to leverage the great new scenarios that OneGet offers!

As usual, happy packaging! :)

Receive feed activity notifications through RSS

Ever wanted to get notified when a new package is available on your feed? Did you know you can just browse the Packages collection on any NuGet feed?


Obviously, that also works on MyGet feeds, but did you know that ever since we launched MyGet, we had this little RSS endpoint sitting around?


In it's early state it was just exposing your feed's OData Packages collection. However, as we introduced activity logs and started collecting this data to show up in activity streams on the web site, we can now also provide you with a more useful RSS feed.

Wouldn't it be useful if we just gave you your activitylog through RSS? Let's see what it looks like in it's current form:

We think this is pretty useful. This is only a minimum-viable notification system in our opinion and we'd love to get feedback on how we can improve this further to match your needs. So please, don't hesitate to reach out and let your voice be heard! One address: http://myget.uservoice.com.

Besides simply subscribing to the RSS feed, you might want to make the internet work for you.

Our friends at MessageHandler have built something that hooks into our activity streams, allowing you to configure a handler that sends you an email and creates a GitHub issue when a MyGet build fails.

For those using IFTTT (if-this-then-that), we've also created a few recipes (or you can build your own):

Happy Packaging!

Checking NuGet package vulnerabilities with OWASP SafeNuGet

Edit - October 14, 2016 - We have a new, integrated vulnerability scan service.

A couple of days ago, OWASP released a new NuGet package which is able to check known vulnerabilities in NuGet packages. Use of libraries with known vulnerabilities can be an issue for software and components you create: check the excellent whitepaper "The Unfortunate Reality of Insecure Libraries". In the OWASP Top 10 2013, consuming vulnerable packages is listed under A9 Using Known Vulnerable Components.

There is a simple solution to this: by installing one additional package in your projects, automatic checking for known vulnerabilities can be done. The SafeNuGet package contains an MSBuild task which will warn you about this.

A repository with vulnerable packages and the reason for that can be found on the SafeNuGet GitHub project. When running a build which references vulnerable NuGet packages, the warnings list will contain some information about this as well as a link with some explanation:

Checking package vulnerabilities

And of course when such library is built using MyGet Build Services, a warning will also be displayed in the build log:

MyGet build services security scan

It would be great if the build would fail entirely when such package is found, right? Well, that is a simple configuration parameter for the SafeNuGet package. Find the SafeNuGet.targets file and update its contents to:

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <UsingTask AssemblyFile="SafeNuGet.dll" TaskName="SafeNuGet.AreNuGetPackagesSafe"  />
  <Target Name="AfterBuild">
    <AreNuGetPackagesSafe ProjectPath="$(MSBuildProjectDirectory)"
         CacheTimeInMinutes="10" DontBreakBuild="false" />
  </Target>
</Project>

Want to make sure known vulnerabilities are shown in your builds? You know the drill:

SafeNuGet

Happy and safe packaging!

Specifying which projects get built with MyGet Build Services

Using MyGet Build Services, you have the opportunity to control exactly how your project gets built. By default, several conventions are used to run builds. MyGet will scan the contents of your Source Control Repository looking for a file which it can work with. In order of precedence, the following files are searched for:

  • Project files (*.csproj, *.vbproj, ...) specified in the build source configuration.
  • MyGet.bat, MyGet.cmd or MyGet.ps1
  • build.bat, build.cmd or build.ps1
  • MyGet.sln
  • Any other *.sln file
  • *.csproj (and *.vbproj, etc)
  • *.nuspec

With the latest deployment of MyGet Build Services, it is now possible to specify which project(s) to build, per build source configuration.

MyGet Specify Project to Build

The projects to build can be C# or VB.NET projects or solutions. Based on the files found, the build process will be slightly different. See the documentation on MyGet Build Services for more information.

Happy packaging!

Migrate away from MSBuild-based NuGet package restore

Back in the days...

NuGet package restore used to be MSBuild-based. You had to explicitly enable it using the context menu on a Visual Studio solution: right-click the solution and select Enable NuGet Package Restore. In fact, if you go to the NuGet docs, you'll see that this scenario is still fully documented. A quick search for "package restore" will throw this old scenario "in your face", as it is the first hit in the search results.

First hit in search results when looking for Package Restore on the NuGet docs

To be fair, the page does highlight that there's a new way of doing this. But many people don't read. At best some look at the pictures. That's why I won't even include a screenshot of that page, as it is full of project setup details that no one should ever go through again. Instead, I'll give you a clear picture of what you should not do :)

Don't do this!

You're doing it wrong!

I can't stress it enough. I'm a huge proponent of NuGet package restore! But if you follow this workflow, then please do it right! (and design for failure, obviously).

The MSBuild-based NuGet package restore has issues. For one: it's MSBuild-based. This means that anything that happens during package restore is run within the MSBuild process, which is particularly annoying for packages that want to modify project files and inject MSBuild targets (as these aren't picked up until the next run).

The moment you manually enable NuGet package restore through the context menu, you're actually installing a few NuGet packages: NuGet.Build, which depends on NuGet.Commandline. The nuget.exe along with a nuget.config and a nuget.targets file are created within a .nuget folder, and all projects that have NuGet package references will be modified to import the NuGet.targets file. The nuget.targets file ensures that nuget.exe is invoked during builds (as in: not before builds!).

The right way

All you need to do is to make sure that your Visual Studio options allow NuGet to download any missing packages in a pre-build phase (note: even before MSBuild compilation starts!). I'm not going to rephrase step-by-step what you should do as David Ebbo already has a great post explaining all of this!

Ensure NuGet is allowed to download missing packages

If you're cloning a new project that did not commit any NuGet packages (and is not using the old MSBuild-based restore), then it just works!

Migrating from the old way

If you still have a .nuget folder in your repository, then please migrate away from it! Think about all those adorable kittens...

Did you know this has been documented on the NuGet Docs all along?! Follow this how-to and save yourself and everyone using your codebase some trouble and follow it step-by-step.

But... my precious (build server)

Here you go: set this environment variable to true and be done with it.

EnableNuGetPackageRestore=true

The following tools support the new automatic package restore out-of-the-box and Just Work™!

The next list of tools requires some minor modifications to the build process:

  • Visual Studio Online / Team Foundation Server (how-to)
  • TeamCity (how-to)

Note that you don't need to worry about development machines! As long as you all have the latest NuGet Visual Studio extension installed.

Upgrading your NuGet extension is generally a good idea anyway, as there are lots of improvements in the latest versions!

Going forward

Here's what I'd love to see happen going forward:

  • The NuGet Docs should by default show the new non-MSBuild-based package restore instructions. There are close to none, but this should be thrown in your face when looking for it.
  • Migration instructions should be clearly linked to.
  • The old MSBuild-based instructions should be archived, perhaps even removed.
  • The context menu-item to manually enable NuGet package restore (MSBuild-based) should be completely removed from the extension. I don't see any reason to keep it. Do you? If you do, please comment on this CodePlex issue, if you agree, then vote for it :)
  • Preferably, the next NuGet Visual Studio extension detects you are using the "old" restore option when you open a solution, and asks you to migrate/upgrade to the new way. Ideally, this removes the targets and import statements, and custom package sources and credentials are taken into account if they are in the nuget.config file.

I'm happy to take on an issue or send PR's for any of the above, but some of the bullet-points seem too big to me to be taken in as a PR.

Which packages are added to a feed during build?

With MyGet Build Services, it is very easy to create NuGet packages from source control. Link a GitHub, BitBucket or CodePlex project to your MyGet feed and we’ll take care of building it and publishing generated packages to that feed. But which packages are added to your feed?

By default, MyGet will add all NuGet packages generated during build to your feed, as long as they are created in a folder named other than packages. The reason for this is that the packages folder is reserved by NuGet itself and may contain packages that were used during the build process and are not necessarily to be added to your feed. When creating a batch-based build, make sure to generate packages in a folder not named packages. A good example folder name could be output.

How to be selective about this? Is it possible to specify which packages are added to your feed? Well yes! To override the default behaviour, a series of wildcard matches can be specified in the build source configuration. When omitted, all packages generated during build will be pushed to your feed. When specified, only packages matching any of the specified package names or wildcards will be pushed to your feed.

image

In the above example, all package names matching Google*.nupkg or Newtonsoft* will be added to your feed.

Happy packaging!

We Love Our Customers!

Whether you are into Valentine's Day or not, we love you! That's why we are offering a 50% discount to all new subscriptions (Starter, Professional) as well as to all subscription renewals (Starter, Professional) purchased on Valentine's Day and the whole weekend after! 

We want you to spend time with the ones you love and care about, and not worry about missing this limited offer. As we are offering our services worldwide, we also don't want to annoy anyone with timezone differences. 

That's why this offer is valid from Friday February 14th (UTC-12) until Sunday February 16th (UTC+12). No matter what timezone you're in, you have the whole 4 days :-)

Checkout the subscription plans and share the love!

Happy packaging!

Default feed endpoint issues 7 February 2013 - Post mortem

On February 7, 2013, we've experienced a partial downtime. The default endpoint for consuming NuGet feeds was slow to respond (+/- 2 minutes for a simple request) and caused issues with a lot of our users and their development teams and processes. We share your pain: we use MyGet for developing MyGet and we hate when things like this happen as much as you do. Our sincerest apologies for the inconvenience caused!

In this post, we'll have a look at what happened and the reason this happened. But before we dive in: we've had 3 hours of partial downtime on the default feed endpoint which most users have configured. We want to apologize by extending all paid subscriptions (Starter, Professional and Enterprise) with one additional month. No action is required on your end, your subscription has already been extended.

What happened?

At around 3:15 PM (CET), our monitoring alerted us about latency on the default feed endpoint going up. After a quick investigation and some additional time, latency went back to normal. One hour later, around 4:15 PM (CET), we saw the same happening again, as well as a number of support e-mails coming in.

MyGet has the following feed endpoints available (see documentation):

  • /F/<your_feed_name> - the NuGet v2 API endpoint
  • /F/<your_feed_name>/api/v2 - the NuGet v2 API endpoint for consuming packages
  • /F/<your_feed_name>/api/v2/package - the NuGet v2 API endpoint for pushing packages
  • /F/<your_feed_name>/api/v1 - the NuGet v1 API endpoint for consuming and pushing packages (still in use by Orchard CMS and some others)

In this case, the default endpoint we offer our users was experiencing issues, the other endpoints were not. We decided to reply all support requests with the advice to switch to the v2 feed endpoint to make sure you could continue work. This requires some reconfiguration, but we figured it's better than having very slow access and timeouts connecting to your feeds.

We checked logs, diagnostics, monitoring reports, read and re-read our code, but could not pinpoint any reason for the default feed endpoint having such high latency. At 5:00 PM (CET) we decided to fail-over to our secondary datacenter. Being a separate environment, we wanted to see if the issue would haoppen there as well. It didn't! So we flipped the DR switch and took our primary datacenter out of the loop to be able to investigate the servers without causing additional issues for our users. This fail-over solved the issue for most of our users, who were able to work with MyGet's default feed endpoint from around 6:00 PM (CET), mind some DNS propagation for some users.

Digging through more logs, we discovered the issue that caused the default feed endpoint to experience this latency (we'll elaborate in a minute). At 8:00 PM (CET), we rolled out a hotfix to our secondary datacenter to make sure the issue would not resurface there either. We worked on a final fix for the next few hours and at 11:30 PM (CET), we flipped the switch back to our primary datacenter.

Why was the default feed endpoint so slow?

As you know, MyGet supports adding upstream feeds and allows to proxy them. This is a very useful feature as one feed can essentially bundle several others, so a developer team only has to configure one and still get packages from multiple.

MyGet treats all upstream package sources as external feeds, even if an upstream feed is a MyGet feed. The reason for that is security contexts can be diferent and we want to have all security aspects applied at any time. This does mean that a MyGet feed having an upstream MyGet feed as a package source is effectively using the default feed endpoint of this second feed, going oer our internal network through the load balancer. Next to security, this also allows us to fan out queries going to multiple package sources over multiple machines.

One of our users configured upstream package sources that were referencing themselves: feed A referencing feed B referencing feed A. We prohibit self-referencing feeds, but the scenario this user tried to achieve was unsupported and resulted in a reference loop on the default feed endpoint. The default feed endpoint was bringing itself down by fanning out queries to itself.

Previous incidents teached us to partition as many things as possible, and luckily we partition all alternative feed endpoints. This allowed us to direct support requests towards the other feed endpoints so users could keep using our service.

Solution

As a quick fix, we disabled the faulting package source so our service could be resumed in a stable fashion. Once that was done, we worked on a permanent solution to this issue. Do we want to block self-referencing feeds? Absolutely, as that makes no sense. Do we want to block adding other MyGet feeds as upstream package sources? No.

We want to keep supporting having upstream package sources that can be MyGet feeds, so that complex hierarchies of feeds and privileges can be configured. Quite a number of feeds do this today, having one feed proxying a series of upstream feeds.

How do we prevent the same problem from happening again? We are now actively tracking refering feeds and detecting loops in these references. Doing this, we keep supporting all scenarios that were possible before while protecting ourselves from shooting ourselves in the foot. Feed A can reference feed B, which can reference feed A. Users can make use of feed A as the entry pont, as well as feed B, and get all expected packages on the feed. We will make sure reference loops are removed from the feed hierarchy.

When a MyGet feed references a non-MyGet feed, we will also be adding an HTTP header to all upstream requests, X-NuGet-Feedchain, which contains the full list of refering feeds. Other NuGet feed implementations can make use of this header to detect reference loops on their end as well.

While we replied to all support requests in under 5 minutes with a workaround to use one of the other endpoints, we will be working on extending our status page at http://status.myget.org, showing details about all endpoints. We also want to be able to post status messages and tips & tricks like switching to the secondary feed endpoints on there.

Hardware fails, software has bugs and people make mistakes. We strive to mitigate as many of these factors and maintain a high quality service with fast, good support. We'll keep doing that to provide you with the best experience possible.

Again, we do apologize for the inconvenience caused.

Happy packaging!