Are actively managed funds still worth owning?

Last year was a rough one for actively managed mutual funds.

Sure, the stock market saw double-digit growth again, and bonds had a surprisingly strong year. But now the taxman cometh, and many mutual fund investors with taxable accounts are finding out that their actively managed funds realized significant capital gains, which they must now pay taxes on. (When a mutual fund sells some of its holdings at a gain, it must pass through the taxable income to shareholders — even if those distributions are then reinvested in the fund.)

Some 453 mutual funds estimated their capital gains distributions for 2014 would be more than 10 percent of net asset value, 52 estimated distributions would exceed 20 percent, and 12 projected distributions of 30 percent or more as of Dec. 15, according to CapGainsValet.com, a website that tracks mutual fund distributions. The site's creator, Mark Wilson, chief investment officer of The Tarbox Group, a California-based wealth management firm, said that by his estimate, "maybe two or three index funds had greater than 10 percent distributions." The other funds were all actively managed.

The larger distributions, and resulting tax bills, from the actively managed funds would be easier to swallow if they were accompanied by extraordinary performance. But a majority of actively managed mutual funds are being outperformed by passive funds, including exchange-traded funds (or ETFs), that mirror popular indexes. About 85 percent of active large-cap stock funds failed to beat or even match their benchmark index through Nov. 25, according to Lipper, a research unit of Thomson Reuters.

Read MoreHoliday spending hangover? Get your finances back on track

Not surprisingly, investors are wondering: Does it make sense anymore to invest in traditional actively managed mutual funds?

"In taxable accounts, actively managed funds are really tough," Wilson said. "It's really difficult to beat indexing."

If recent inflow and outflow patterns are any indication, a growing number of investors agree.

In December, fund-research form Morningstar reported that in the previous year, active U.S. equity funds lost $91.9 billion in outflows while passive U.S. equity funds drew in $156.1 billion. "A clear pattern has emerged this year," wrote senior analyst Alina Tarlea, "consistent outflows on the active side and inflows on the passive side."

One beneficiary has been Vanguard: On Sunday, The Wall Street Journal reported that investors poured $216 billion into the biggest provider of index-tracking products, a record inflow for any mutual-fund firm.

ETFs as a category are also benefiting. The number of ETFs, which trade like stocks but hold a basket of assets like a mutual fund, grew from 113 in 2002 to nearly 1,300 in 2013, according to the Investment Company Institute. And last month, they hit another milestone, surpassing $2 trillion in assets. While that's still dwarfed by the $16 trillion in mutual funds, ETFs are growing at a faster rate. In 2010, they had just half the assets they do now.

Actively managed funds have another problem when compared to index-fund investing and ETFs: in general, their fees are higher. While fees on all funds have come down over the years, average fees on actively managed equity funds were 89 basis points in 2013, compared with 12 basis points for index equity funds, according to the Investment Company Institute. (When it comes to the cost of index funds relative to the cost of ETFs, a study by Alex Bryan and Michael Rawson of Morningstar found that "the difference in expenses between the two vehicles is small.")

Still, some active fund managers out there do beat the market. One example: Morningstar has nominated the managers of several domestic stock funds, including the American Century Mid Cap Value Fund and T. Rowe Price Mid Cap Growth, as contenders for domestic stock fund manager of the year. These funds outpaced the vast majority of their peers and beat their benchmark indices.

"It's hard work separating the wheat from the chaff" in the world of actively managed funds, said Stephen Horan, managing director at the CFA Institute, but winners do exist.

Read MoreHow many 401(k) choices are too many?

John Rekenthaler, vice president of research at Morningstar, examined the total returns of different fund management approaches in 2014, comparing actively managed Vanguard funds, which have relatively low fees, to other low-cost active funds, Vanguard passively managed funds, and actively managed funds with high fees. The highest ranked were Vanguard's actively managed funds, followed by the other low-cost active funds and the passive funds. The high-cost actively managed funds were the lowest ranked.

"It's true that active management sold at its customarily steep price is second rate. But so are high-cost index funds," he concluded. "How a fund is managed is less important than its cost headwind."

As for tax efficiency, exchange-traded funds and index funds both tend to generate fewer capital gains than actively managed funds year by year, partly because they buy and sell securities less often. ETFs in particular have an advantage because managers create or redeem "creation units" rather than actual securities to manage inflows and outflows or to rejigger asset allocation. But when investors sell their ETFs, index funds, or actively managed mutual funds, they owe capital gains on any appreciation.

Read MoreInvestors flocking into index funds – here's why

"ETFs and mutual funds have to follow the exact same tax rules," said Joel Dickson, principal in the investment strategy group at Vanguard.

Some experts say many of the tax costs of mutual fund investing simply occur at different times than the costs associated with investing with ETFs. Horan points out that when a mutual fund distributes capital gains, its net asset value decreases by that amount. So all else being equal, when an investor sells the mutual fund shares, the gain logged then will be smaller than it would have been without the distribution.

The mechanics of buying and selling mutual funds and ETFs are another consideration. ETFs can be traded any time the markets are open, while mutual fund sellers and buyers get the end-of-day price no matter when they put in the order. But when an investor buys or sells a mutual fund position, the price is simply the end-of-day price, while the seller of an ETF position may incur an additional, unseen cost if the ETF trades at a discount to the underlying assets, or if it is trading at a premium when the investor buys in.

"Usually the value of the ETF is very close to the underlying holdings, but not always," Dickson said.

(Closed-end mutual funds also trade at a premium or discount to net asset value, but at $279 billion at the end of 2013, assets in closed-end funds were a tiny fraction of the $15 trillion in U.S. mutual fund assets, according to the Investment Company Institute, and about one-sixth of those in ETFs.)

Horan argues that the ability to trade ETFs anytime may actually hurt average investors. "Having the ability to trade intraday can actually work against you in a behavioral, psychological way. There is plenty of evidence to suggest that the more individual investors trade, the worse they do," he said.

One example: Terrance Odean, a professor at University of California Berkeley's Haas School of Business, examined trading records from a large discount brokerage firm, and said in a 2014 interview that "both men and women on average underperformed the buy-and-hold approach to investing." (In an interesting side note, he also found that men underperformed by one percentage point more a year than women.)

Read MoreWill new pension guarantees benefit you?

Ultimately, Horan said, investors should first determine their investment approach. The type of investment will follow from that.

"Index funds are a good long-term strategy for folks that don't necessarily have an expertise in picking managers, or picking stocks themselves. That's going to be the majority of us," he said. "Are there active managers that have key insights and a steely discipline and an ability to capitalize and execute on those insights? I think the answer is yes. But they are not the norm."